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dissemination  to  Army  users.  A  separate  Operations  Handbook  was  issued  for  the 
computer-aided  APM  demonstration  system. 
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I.  EXECUTIVE  SUMMARY 


This  report  describes  an  analytic  process  model  (APM)  for  systems  design  and 
measurement.  This  model  provides  a  uniform,  thorough,  adaptive  and  efficient 
procedure  to  derive  design  specifications  and  effectiveness  measures  for  any 
human-machine  system,  including  any  training  system. 

The  APM  is  designed  to  enable  testers,  analysts  and  researchers  to  define  sets 
(or  taxonomies)  of  system  performance  requirements,  and  to  translate  then  into 
measures  and/or  design  requirements.  It  forces  one  to  describe  the  system  of 
interest  and  its  human  elements  in  terms  of  a  more  complete  set  of 
system-human  attributes  pertinent  to  system  effectiveness.  The  general  APM 
is  illustrated  in  Figure  1.  An  interactive  demonstration  was  developed  to 
illustrate  the  potential  for  a  computer-aided  APM  to  generate  effectiveness 
measures  (Figure  1,  Blocks  1,  2  and  4)  for  training  systems  in  particular.  It 
provides  general  "menus"  of  performance  items  and  procedures  to  help  trans¬ 
late  those  items  into  appropriate  evaluation  measures.  Work  with  this 
demonstration  computer-aided  model  suggests  that  similar  routinized  programs 
can  also  be  developed  for  other  particular  kinds  of  systems  (e.g.,  com¬ 
munication,  transportation,  weapon,  etc.). 

This  project  addresses  the  problem  that  testers,  analysts  and  researchers  too 
often  use  an  incorrect,  incomplete  or  inappropriate  set  of  human  performance 
factors  in  evaluating  or  specifying  a  human-machine  (e.g.,  training)  system. 
Those  are  usually  known  measures  and  traditional  design  choices,  often 
selected  without  adequate  consideration  of  the  system  qua  system.  They  may 
not  ask  the  essential  questions  about  system  performance  and  effectiveness  or 
they  may  provide  inadequate  answers  to  these  essential  questions..  Because 
no  verified  analytic  process  for  deriving  (or  specifying)  the  optimal  measures 
of  a  system's  performance  or  effectiveness  exists,  true  assessment  needs  are 
difficult  to  define  and  the  process  is  relatively  haphazard.  The  typical 
solution  is  to  test/measure/specify  the  easy  and  accessible  system  points,  but 
not  necessarily  those  that  should  be  addressed.  Without  having  more  sys¬ 
tematic  procedures,  people  measure  or  specify  what  can  be  counted  (e.g.,  POI 
hours),  or  observed  (e.g.,  number  of  troops  trained).  They  design  written 
tests  to  assess  the  learners'  rote  memories  rather  than  their  understanding, 
and  use  criteria  such  as  end-of-course  tests  rather  than  on-the-job  perfor¬ 
mance.  Rarely  do  people  know  the  relationship  between  test  performance  and 
job  performance,  or  between  soldier  job  performance  and  unit  effectiveness. 
This  results  in  wasting  valuable  resources  (time,  effort,  talent,  money), 
failing  to  provide  adequate  answers  to  effectiveness  questions,  and  losing 
important  facts  regarding  human  contributions  to  system  performance.  The 
APM  is  designed  to  be  a  better  method  for  deciding  what  should  be  measured, 
or  designed,  and  how. 

The  recent  development  and  tryout  of  the  APM  were  performed  within  the 
context  of  the  Bradley  Infantry  Fighting  Vehicle  (BIFV)  development  program, 
a  complex  man-machine  and  training  system.  The  following  products  have 
resulted:  an  outline  APM  (Figure  1);  a  de. ailed  STM  (Figure  1,  Block  1)  and 
associated  procedures  for  deriving  dimensions  or  taxa  of  system  performance; 
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Figure  1.  Overview  of  the  APM  for  System  Design  and  Evaluation 
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initial  procedures  for  deriving  attributes  (Figure  1,  Block  2),  design 
specifications  (Figure  1,  Block  3)  and  measures  (Figure  1,  Block  4)  from  the 
taxa;  identification  of  taxa  for  the  surveillance  function  of  the  BIFV;  definition 
of  what  functional  subsystems  constitute  a  training  system;  partial  or,  in  some 
cases,  complete  identification  of  taxa  for  the  subsystems  of  the  BIFV  training 
system  (BIFVTS);  and  a  demonstration  computer-aided  APM  program. 
Although  the  sample  taxa  have  been  developed  for  the  BIFV,  they  appear  to 
be  generic  and  applicable  to  any  manned  system. 

I’he  APM  application  is  illustrated  in  Figure  2.  This  shows  how  an  analyst 
could  produce  effectiveness  measures  using  the  computer-aided  model.  The 
procedure  consists  of  branching  through  a  series  of  nested  sets.  The  analyst 
chooses  from  the  computer's  data  base  of  recommended  options  or  creates  new 
options  at  each  branch.  Special  instructions  and  help  are  available  on-line  and 
in  an  operating  manual  to  guide  the  analyst  through  the  model.  Hard-copy 
printouts  of  the  selected  options  are  available  immediately  upon  request  at  key 
stages  of  the  analytic  process.  Figure  3  is  a  sample  of  such  a  printout, 
including  a  portion  of  the  Figure  2  illustration.  The  sequential  digits  in  the 
index  numbers  of  Figure  3  correspond  to  the  analytic  items  and  levels  illus¬ 
trated  in  Figure  2  (Aspects,  Objectives,  Functional  Purposes,  Characteristics, 
Attributes  and  Measures,  respectively).  As  seen  in  Figure  3,  attributes  and 
measures  can  be  derived  from  any  of  the  three  analytic  levels  (Objectives, 
Functional  Purposes  or  Characteristics).  Consequently,  zeros  can  appear  in 
the  third  (Functional  Purposes)  or  fourth  (Characteristics)  positions  of  the 
index  number.  The  header  information  and  index  numbers  provide  a  means 
for  tracing  the  connection  between  any  selected  performance  measure  and  its 
determining  prerequisites.  One  can  thereby  insure  that  all  system  objectives 
are  addressed  by  derivative  evaluation  measures  and  that  all  measures  are 
justified. 

As  described  in  Chapter  IV,  the  APM  compares  in  several  important  ways  with 
other  models  for  evaluating  or  developing  training  systems  and  other 
human-machine  systems.  For  example,  it  is  totally  consistent  with  the  general 
objectives  of  MIL-I1-46855B  ("Hunan  Engineering  Requirements  for  Military 
Systems,  Equipment  and  Facilities"),  by  providing  a  general,  sequential  method 
by  which  human-machine  systems  can  be  developed  in  a  total  systems  context 
and  with  careful  emphasis  on  the  human  element.  It  complements  a  variety  of 
methods  grouped  under  the  generic  term  Instructional  System  Development 
(ISD)  by  facilitating  the  front-end  analysis  of  training  systems.  ISD 
frequently  comes  too  late  to  identify  some  potential  training  problems,  whereas 
the  APM  can  be  applied  earlier  when  such  issues  as  performance  goals, 
mission,  objectives,  functions,  tasks,  job  analysis,  validation  and  logistic*? 
must  be  first  addressed.  The  broader  and  earlier  analysis  provided  by  the 
APM  can  set  the  stage  for  the  detailed  processes  associated  with  ISD.  The 
APM  is  also  complementary  to  the  Early  Training  Estimation  System  (ETES). 
The  APM  is  of  special  value  in  facilitating  the  generation  of  system  and 
subsystem  objectives  and  the  development  of  performance  measures,  while  ETFS 
is  of  particular  value  in  evaluations  of  media  and  instructional  methods.  The 
APM  is  compatible  with  various  methods  for  Logistic  Support  Analysis  (LSA), 
in  that  earlier  estimates  of  logistics  requirements  can  be  prepared  with  the  aid 
of  the  APM.  Because  the  APM  can  be  applied  to  developing  systems  in  which 
operating  parameters  are  still  evolving,  it  also  can  be  useful  for  generating 
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1.2.  The  subsystem  must  be  capable  of:  establishing  performance 
objectives. 

_j_l - T - , - , - | - , - 

1.2.1.  This  subsystem  capability  allows:  stating  expected  performance 
and  behavior  after  training. 

~ ^ - 1 - 1 - 1 - 1 - "~l - 

1.21.1.  For  this  purpose,  the  system  must  have  the  potential  for: 
defining  the  performance  action. 
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1.2.1. 1.1.  To  evaluate  effectiveness  in  meeting  this  characteristic,  the 
following  system  attribute(s)  can  be  measured:  types  of  BIFV 
gunner  performance  actions  intended  to  be  defined. 

~ | - I - 1 - 1 - 1 - 1 - 1 - 

1.2.1. 1.1.1.  This  system  attribute  can  be  analyzed  by  comparing  the 
followir^  parameter(s)  with  established  criteria:  relevant 
types  of  actions  not  intended  to  be  defined. 
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1.2. 0.0. 3. 3.  Inconsistencies/deficiencies  in  sequencing  steps  in  plons 
1.2. 0.0. 3. 4.  Reviewers  cum  ratings  of  soundness/workability  of  plans 
1.2. 0.0. 4.0.  Resources  allocated  to  establishing  DIFVG  performance  objectives 
1.2. 0.0. 4.1.  X  of  reqd  person-hrs  proposed  for  alloc  to  establish  objectives 
1.2. 0.0. 4. 2.  X  of  reqd  materiols/goods  proposed  for  alloc  to  establish  objetvs 
1.2. 0.0. 4. 3.  X  of  reqd  support  services  proposed  for  alloc  to  establish  objetvs 
1.2. 0.0. 4. 4.  Ratio  of  *  proposed  to  I  needed  for  establishing  objectives 
1.2. 1.0. 0.0.  Stating  expected  performance  and  behovior  after  training 

1.2. 1.0. 1.0.  Elements  or  factors  of  gunner  abilities  intended  to  be  stated 
1.2. 1.0. 1.1.  Essential  elements  of  abilities  to  be  incl  in  stmnt  of  DIFVG  oblty 


Figure  3.  Sample  Printout  of  Analysis  Results 
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early  inputs  (objectives,  parameters,  measures)  to  the  Army's  Human 
Resources  Test  and  Evaluation  System  (HRTES).  While  the  APM  provides  a 
measurement  framework  for  the  HRTES  technique,  HRTES  extends  APM  further 
into  the  operational  testing  domain. 

Now  that  a  computer-aided  demonstration  system  of  the  APM  is  available, 
Chapter  V  considers  how  it  can  be  improved.  The  demonstration  system  can 
now  be  used  to  develop  evaluation  measures  for  three  of  the  six  training 
subsystems:  Design,  Enabling  and  Delivery,  but  not  Command,  Logistics  or 

Emplacement  (see  Appendices  A,  B  and  C).  It  is  essential  to  develop  the  taxa 
of  subsystem  performance  items,  subsystem  attributes  and  attribute  measures 
for  the  three  remaining  training  subsystems  to  use  the  APM  fully  for 
evaluation.  Procedural  guidelines  must  be  developed  for  generating  design 
specifications  from  the  performance  taxa  to  use  it  in  developing  design 
specifications.  More  complete  computer  "help"  instructions  must  be  made 
available  on  request.  Eventually,  the  model  must  include  data  bases  (taxa)  for 
more  than  training  systems:  computer  assistance  should  be  tailored  to 
communications  systems,  weapon  systems,  transportation  systems,  and  many 
others  of  interest  to  particular  users.  The  package  could  also  incorporate  a 
routine  for  costing  guidelines  to  aid  in  early  evaluation  decisions.  Clearly,  if 
these  capabilities  are  added  to  the  computer-aided  APM,  it  will  require  a 
computer  of  significantly  greater  capacity  than  the  present  Apple  II  Plus  which 
barely  contains  the  sample  demonstration  program. 
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II.  TIIL  ANALYTIC  PROCESS  MODEL 


The  APM  is  a  conceptual  framework  for  systematizing  the  process  of  designing 
and  evaluating  human-machine  systems.  Impetus  for  APM  development  is  the 
experience  that  system  design  and  evaluation  often  proceed  haphazardly. 
Design  and  measurement  decisions  are  frequently  formulated  out  of  familiar 
practice,  without  due  consid  ;ration  of  new  approaches  that  may  be  preferable. 
Conversely,  designers  and  evaluators  sometimes  adopt  emerging  technology  or 
methods  because  they  are  no 'el,  without  adequate  regard  for  what  is  best 
suited  to  the  application  at  hand.  In  either  case,  the  system  may  be  built 
and/or  tested  incorrectly,  incompletely  or  inappropriately.  This  results  in 
misleading  evaluative  findings,  a  .d  systems  that  do  not  work  as  well  as  they 
should . 


The  fundamental  concept  behind  the  APM  is  that  the  haphazard  design  and 
evaluation  typically  begin  with  a  failure  to  consider  fully  the  system's  intended 
work .  That  is,  designers  often  start  to  build  systems,  and  evaluators  often 
start  to  test  them,  without  a  clear  understanding  of  what  the  systems  are 
suppose!  to  do,  and  where,  when,  why  and  how  they  are  supposed  to  do  it. 
The  APM  ic  designed  to  remind  (or  force)  designers  and  evaluators  to  consider 
all  relevant  factors. 


The  APM  Concept  and  Definitions 


Based  on  the  original  work  of  Finley  et  al.  (1975,  1976)  and  extended  by 
Bloom  et  al.  (1981,  1982),  a  four-stage  functional  sequence  was  defined  as  the 
top  level  (or  macroscopic)  outline  of  the  APM.  This  is  illustrated  in  Figure  4, 
with  a  cross-reference  to  the  blocks  of  Figure  1. 


Stage  1,  the  contextual  stage,  is  the  Systems  Taxonomy  Model  (STM)  as 
initially  defined  by  Finley  et  al.  (1975).  It  is  an  organized  conceptual 
framework  and  a  set  of  data  bases  and  guidelines  that  assist  an  analyst  in 
determining  what  the  system  of  interest  is  supposed  to  do  and  why,  how, 
where  and  under  what  circumstances  it  is  supposed  to  do  it.  The  STM  is  a 
model  for  taxonomization  since  it  provides  a  means  of  generating  dimensions 
which  would  be  taxa  in  that  they  could  be  used  for  systems  classification  and 
as  generic  terms  for  analysis  to  develop  measures  of  effectiveness  and 
performance.  The  STM  has  three  dimensions: 


t.  >pects  of  system  performance  (of  which  there  are  five). 

Levels  of  system  description  (of  which  there  are  three). 

System  hierarchical  structure  (which  can  contain  any  number  of 
suprasystems,  subsystems  and  collateral  systems). 


Stage  2,  the  focal  stage,  includes  analytic  procedures  for  translating  the 
required  work  into  corresponding  system  attributes.  This  stage  also  includes 
procedures  for  determining  how  to  insure  the  presence  of  those  attributes,  and 
for  determining  whether  they  are  present  to  the  appropriate  degree.  The 
outputs  of  Stage  2  are  the  basic  tools  of  a  system  evaluation  application 
(measures  of  performance/effectiveness)  and/or  of  a  system  design  application 
(design  specifications). 


^  jJiUMJiWJMi.uiu 


Corresponding  Blocks 
in  Figure  1 


Figure  4.  Four  Stages  of  the  APM 
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Stage  3,  the  implementation  stage,  consists  of  methodology  for  employing 
the  application  tools.  For  a  design  application,  Stage  3  is  where  the  system  is 
actually  built,  in  accordance  with  specifications.  For  an  evaluation  application. 
Stage  3  is  where  the  system  is  actually  tested,  using  the  measures. 

Stage  4,  the  interpretation  stage,  includes  analytic  procedures  for 
determining  whether  the  system  satisfies  its  work  requirements  and,  when 
deficiencies  exist,  for  determining  how  the  system  must  be  modified  to  correct 
the  deficiencies. 

To  date,  the  key  concepts  have  been  developed  for  Stages  1  and  3. 
They  are  described  in  the  following  paragraphs. 


1 .  Aspects  of  System  Performance 

A  system's  work  is  broadly  defined  by  what  it  is  supposed  to  do  and 
where  it  is  supposed  to  do  it.  To  support  design  or  evaluation,  the  "what"  of 
performance  can  be  divided  into  three  mutually  exclusive  classes,  or  aspects: 

c  Performance  potentialities — the  capabilities  for  doing  work  that 
the  system  is  supposed  to  possess; 

°  Performance  processes-~thc  .activities  that  the  system  is 
supposed  to  carry  out; 

0  Performance  products — the  outputs  that  the  system  is  supposed 
to  deliver. 

A  system  could  fail  at  any  of  these  three  aspects  of  performance.  It 
could  simply  not  have  the  prerequisite  abilities  needed  to  get  its  job  done;  it 
might  have  the  ability,  but  fail  to  implement  the  necessary  procedures;  it 
might  have  the  ability  and  carry  out  the  procedures,  but  fail  to  deliver  the 
required  goods  and  services  in  the  quantity  and  quality  neoded.  Too  often, 
system  evaluators  concern  themselves  only  with  the  output,  or  product,  aspect 
of  system  performance.  But  if  the  product  is  deficient,  no  corrective  action 
can  be  taken  unless  the  evaluation  can  diagnose  the  process  and  potential 
performance  to  uncover  the  root  cause  of  the  failure. 

The  "where"  of  performance  similarly  subdivides  into  two  mutually 
exclusive  aspects: 

°  Performance  environment — the  naturally  occurring  set  of 

conditions  and  circumstances  under  which  the  system  must 
provide  its  capabilities,  carry  out  its  processes,  and  deliver  its 
products. 

0  Performance  constraints--the  man-made  or  artificial  limiting 
factors  that  apply  to  the  system's  potential,  process  and 
product  performance. 

System  designers  and  evaluators  always  must  consider  the  real  world,  where 
the  system  actually  will  be  used.  It  isn't  good  enough  that  a  system  can 
provide  its  required  abilities,  carry  out  its  processes  and  deliver  its  products 


I 
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under  the  most  favorable  circumstances,  or  without  regard  for  cost,  safety, 
time,  or  other  considerations.  Performance  under  actual  operational  conditions 
and  in  full  compliance  with  all  necessary  restrictions  is  demanded  of  every 
system . 

2 .  Levels  of  System  Description 

Three  levels  of  system  description  are  defined: 

0  Objectives 

0  Functional  Purposes 

Characteristics 

The  "what"  and  "where"  of  system  performance  provide  a  sterile  or 
abstract  view  of  a  system's  work.  It  is  a  view  of  work  as  something  that 
merely  is,  without  regard  for  its  purpose.  This  is  the  Objectives -level  of 
work  definition,  i.e.,  the  performance  elements  are  defined  as  objects  or  ends 
in  themselves.  Quite  different  types  of  systems  may  have  similar  work 
requirements  at  the  objectives-level  of  description.  The  process  known  as 
"Task  Analysis,"  for  example,  is  required  in:  tool  design,  systems,  training 
development  systems,  personnel  evaluation  and  management  systems,  and  many 
others . 


But  systems  do  not  perform  work  merely  for  its  own  sake.  There 
always  is  a  purpose  behind  a  system's  work,  a  function  to  which  each  work 
element  is  to  be  applied.  Otherwise,  the  work  really  is  not  required. 
Furthermore,  if  the  system  works  to  accomplish  purposes,  there  must  be  a 
method  to  its  work.  That  is,  it  must  approach  its  work  requirements  in 
characteristic  ways  which  insure  that  the  purposes  will  be  met.  These 
observations  lead  to  definition  of  two  other  levels  of  system  description.  The 
Functional  Purposes-level  corresponds  to  the  "why"  of  performance:  the 
reasons  For  the  existence  of  objectives-level  work  requirements.  The 
Characteristics-level  corresponds  to  the  "how"  of  performance:  the 

characteristic  ways  in  which  the  system  provides  its  potentialities,  carries  out 
its  processes,  and  delivers  its  products  to  insure  that  their  purposes  are  met. 

3 .  Interaction  of  the  First  Two  Dimensions:  Taxa  of  System 
Performance 


The  five  aspects  of  system  performance  interact  with  the  three  levels 
of  system  description  to  form  a  fifteen-cell  matrix.  Each  cell  represents  a 
particular  taxonomy  of  work  requirements,  each  member  of  which  is  a 
particular  taxon  or  dimension  of  work.  As  an  example,  Figure  5  arrays  the 
work  requirements  taxonomies  for  the  Carrier  Team  Subsystem  of  the  Bradley 
Infantry  Fighting  Vehicle,  and  exhibits  sample  member  taxa  for  each  cell. 

Within  each  column  in  Figure  5,  the  member  taxa  are  hierarchically 
linked  from  cell  to  cell.  In  the  topmost  cells,  general  statements  of  the  "what" 
and  "where"  of  performance  are  found.  For  example,  the  system  is  supposed 
to  have  the  potential  for  "providing  surveillance"  (taxon  1.1),  and  is  supposed 
to  do  so  in  a  "rolling  terrain"  environment  (taxon  4.1).  In  the  cells  of  the 
middle  row,  the  purposes  behind  the  required  work  are  stated.  The  reasons 
for  the  potential  of  "providing  surveillance"  include  having  the  capabilities  of 
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ASPECTS  OF  SYSTEM  PERFORMANCE 
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Figure  5.  Sample  2-Dimensional  STM  for  the  Bradley  Infantry 
Fighting  Vehicle  (Carrier  Team  Subsystem) 


"conducting  sector  surveillance"  (taxon  1.1.1),  "detecting  targets"  (1.1.2), 
"acquiring  targets"  (1.1.3),  and  possibly  other  purposes.  Further,  the 
systen  is  required  to  be  able  to  achieve  those  purposes  while  operating  on 
hills  (4.1.1)  and  in  valleys  (4.1.2).  The  decimal  numeric  notation  links  the 
various  functional-purposes-level  taxa  with  the  individual  objectives-level  taxon 
they  support.  Taxa  1.1.1,  1.1.2,  and  1.1.3  all  support  or  "descend  from" 
taxon  1.1. 


The  cells  of  the  bottom  row  state  how  each  purpose  is  to  be 
satisfied.  For  example,  the  capability  of  "detecting  targets"  will  be  met  by 
providing  the  potentials  for  "unaided  visual  target  detection"  (1. 1.2.1),  "aided 
visual  target  detection"  (1.1. 2. 2),  "infrared  sensor  target  detection" 
(1.1.2. 3),  and  perhaps  through  other  capabilities  as  well.  The  decimal 
notation  traces  the  vertical  path  from  cell  to  cell.  Thus,  characteristics-levcl 
taxon  1.1. 2. 3  descends  from  functional-purposes  taxon  1.1.2,  which  in  turn 
descends  from  objectives  taxon  1.1. 

As  Figure  5  illustrates,  the  matrix  resulting  from  the  interaction  of 
the  first  two  dimensions  of  the  STM  provides  a  framework  for  organizing: 

The  basic  work  requirements  for  which  the  systen  is  intended, 
including  its  potential,  process  and  product  aspects  and  the 
natural  and  man-made  factors  impeding  those  aspects; 

°  The  purposes  behind  each  work  requirement,  and  the 

environmental  and  constraining  components  that  bear  on  those 
purposes; 

°  The  ways  in  which  the  system  is  supposed  to  achieve  its 
purposes,  and  the  factors  that  may  impede  those  ways. 

4 .  System  Hierarchical  Structure:  The  Third  Dimension  of  the  STM 

Systems  seldom  work  in  isolation.  Usually  they  interact  with  other 
systems,  so  that  the  work  they  do  influences,  and  is  affected  by,  what  other 
systems  do.  Designers  and  evaluators  need  to  account  for  these  other 
interacting  systems  so  that  all  of  the  requirements  of  the  system  of  interest 
may  be  identified  and  so  that  appropriate  interfaces  may  be  designed  and 
appropriate  evaluation  controls  may  be  established.  Three  kinds  of  interacting 
systems  can  exist: 

Larger  systems,  of  which  the  system  of  interest  is  a  part,  and 
superior  systems  exercising  command  and  control  over  the 
system  of  interest.  Collectively,  these  larger,  superior  systems 
are  termed  suprasystems.  It  is  often  at  the  suprasystem  level 
that  the  work  requirements  of  a  system  of  interest  are 
established. 

Smaller  systems  wholly  contained  within  the  system  of  interest, 
and  systems  over  which  the  system  of  interest  exercises 
command  and  control.  Such  smaller,  subservient  entities  are 
called  subsystems.  The  ability  of  a  system  to  satisfy  its  work 
requirements  o7ten  depends  on  the  subsystems'  abilities  to 
perform  as  they  are  supposed  to  do. 
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°  Systems  that  exist  and  operate  on  the  same  level  of  command 
and  control  as  does  the  system  of  interest,  and  with  which  the 
system  of  interest  shares  resources  or  exchanges  inputs  and 
outputs.  These  equal  ranking,  interacting  systems  are  termed 
collateral  systems.  A  system  of  interest  must  be  designed  to 
interface  compatibly  with  its  collateral  systems. 

Figure  6  illustrates  the  concept  of  system  hierarchical  structure  in  the  context 
of  the  Bradley  Infantry  Fighting  Vehicle.  The  BIFV  is  shown  as  subservient 
to  one  suprnsystcm ,  namely,  the  mechanized  infantry  platoon  of  which  the 
individual  vehicle  system  is  a  member  squad.  It  has  two  subsystems ,  namely, 
the  Carrier  Team  Unit  (Track  Commander,  Gunner,  Driver,  the  vehicle  control 
apparatus,  the  vehicle-mounted  weapons,  etc.),  and  the  Crew  Compartment 
Team  (Firing  port  weapons  and  firing  port  weapon  operators).  Four  collateral 
systems  are  shown.  Three  of  these  are  the  other  individual  fighting- vehicle- 
systems  that  make  up  the  platoon  (including  the  particular  vehicle  that 
happens  to  house  the  platoon  headquarters).  Another  collateral  system  is  the 
enemy  armored-vehicle  system,  i.e.,  the  threat.  In  the  operational 
environment,  the  BIFV  derives  its  work  requirements  from  the  platoon's 
orders,  translates  those  requirements  into  the  corresponding  responsibilities  of 
its  two  subsystems,  and  carries  out  its  duties  in  concert  with  (and  without 
interfering  with)  its  sister  squads.  And  undeniably,  it  carries  out  its  duties 
in  concert  with  the  threat  system,  since  those  duties  include  detecting, 
acquiring,  identifying,  tracking,  engaging,  and  neutralizing  or  destroying  the 
threat.  The  relationship  between  the  BIFV  and  the  threat  is  certainly 
antagonistic,  but  equally  certainly  interactive  and  thus  collateral.  If  the  BIFV 
is  not  designed  with  the  threat's  capabilities  and  work  requirements  in  mind, 
the  BIFV  likely  will  fail  to  achieve  its  purposes.  In  an  engagement  situation, 
the  outcome  may  depend  not  only  on  how  well  the  BIFV  performs  its  intended 
work,  but  also  on  how  well  the  threat  does  its. 

5.  Summary  of  the  3-Dimensional  STM 

To  recapitulate  briefly,  the  STM  is  the  first  stage  of  the  APM.  It  is 
the  stage  at  which  a  svstem's  work  requirements  are  identified  and  organized 
so  that  they  will  be  fully  considered  in  the  design  and  evaluation  of  that 
system.  Identification  jf  the  work  requirements  demands  that  the  analyst 
examine  not  only  the  system  itself,  but  also  its  operational  environment, 
practical  constraints,  ird  the  performance  factors  of  other  systems  with  which 
it  interacts.  In  organizing  "what"  work  the  system  is  supposed  to  accomplish, 
it  is  necessary  to  consider  not  only  the  products,  goods  or  services  to  be 
delivered,  but  also  the  processes  intended  to  lead  to  those  products  and  the 
potentialities  needed  to  carry  out  those  processes  effectively.  Further,  for 
each  element  of  required  work,  it  is  necessary  to  identify  all  of  the  purposes, 
or  reasons,  why  the  work  is  needed,  and  exactly  how  those  purposes  are  to 
be  achieved.  Finally,  each  system  of  interest  must  be  examined  in  the  context 
of  the  hierarchical  system  structure  in  which  it  is  necessarily  embedded — the 
supra-,  sub-  and  collateral  systems. 

6.  Guidelines  for  Identifying  P!mensions/Taxa  of  System  Performance 

Through  experience  gained  in  a  series  of  trial  applications  of  the 
APM  concept,  guidelines  have  evolved  for  identifying  a  system's  work 
requirements  within  each  cell  representing  interaction  of  a  particular  aspect  of 
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Figure  8.  Example  of  a  System  Hierarchical  Structure 
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performance  with  a  particular  level  of  description.  The  guidelines  are  given  in 
Figure  7.  Each  cell's  guidelines  begin  with  the  opening  phrases  of  a  statement 
that  sets  the  context  for  the  particular  cell,  and  that  establishes  the  format 
for  the  member  taxa  or  dimension  of  that  cell.  A  list  of  questions  follows, 
each  of  which  is  intended  to  establish  an  analytic  perspective  that  will  aid  the 
identification  of  those  member  taxa.  The  questions  are  not  intended  to  be 
mutually  exclusive.  Neither  do  they  necessarily  constitute  all  of  the  analytic 
perspectives  that  might  help  in  the  identification  of  the  system's  work 
requirements.  Rather,  they  reflect  conceptual  and  organizational  points  of 
view  that  evolved  from  early  trial  applications  of  the  emerging  STM/APM,  and 
that  proved  helpful  for  identifying  taxa  in  subsequent  applications. 

7 .  Guidelines  for  Deriving  Measurable  System  Attributes 

Application  of  the  guidelines  discussed  above  (Figure  7)  allows  an 
analyst  to  identify  and  organize  a  system's  work  requirements  (Figure  1, 
Block  1).  Those  work  requirements  establish  the  context  for  system  design 
and  system  evaluation.  The  designer  examines  the  work  requirements  and 
asks,  "How  can  I  build  the  system  so  that  it  satisfies  the  work  requirements?" 
The  evaluator  examines  those  same  requirements  and  asks,  "How  can  I  test  the 
system  to  determine  whether  it  satisfies  the  work  requirements?"  The  designer 
must  translate  the  work  requirements  into  concrete  design  decisions,  i.e., 
consciously  control  the  physical  features  of  the  system  so  that  adequate 
provision  is  made  for  the  system  to  work  as  it  should.  The  evaluator  must 
translate  the  work  requirements  into  a  concrete  set  of  measures  that  can  be 
applied  to  observable  features  of  the  system  to  determine  whether  the  system 
can  work  as  it  should,  is  working  as  it  should,  or  has  worked  as  it  should. 

The  translation  of  work  requirements  allows  the  designer  and  the 
evaluator  to  focus  on  the  system's  ATTRIBUTES  (Figure  1,  Block  2).  An 
attribute  is  a  concrete,  observable  feature  of  the  system  that  bears  directly  on 
the  work  that  the  system  can  and  does  perform.  To  the  designer,  an 
attribute  is  something  that  can  be  controlled.  By  controlling  the  attributes, 
the  designer  controls  what  the  system  can  do  and  how  well  it  can  do  it.  To 
the  evaluator,  an  attribute  is  something  that  is  measurable.  By  measuring  the 
attributes,  the  evaluator  arrives  at  a  judgment  or  appraisal  of  the  system's 
work . 


Figure  8  illustrates  the  translation  of  work  requirements  into 
attributes.  The  requirements  chosen  for  the  example  are  a  sequence  of  the 
performance  processes  required  of  the  Design  (or  Curriculum  Development) 
subsystem  of  the  Bradley  Infantry  Fighting  Vehicle  Gunnery  Training  System. 
The  selected  sequence  consists  of  one  objectives-level  performance  process 
(Analysis  of  Tasks),  together  with  one  of  its  functional  purposes,  and  one 
characteristic  of  that  functional  purpose. 

In  order  to  build  the  system  to  carry  out  the  "Analysis  of  Tasks" 
requirement  properly,  the  training  system  designer  must  choose  some 
appropriate  analytic  methods;  insure  that  those  methods  address  the  purposes 
of  the  analysis  (e.g.,  classification  of  domains  of  learning)  and  the  technical 
details  required  to  achieve  those  purposes  (e.g.,  identification  of  physical  skill 
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Figure  7.  Guidelines  for  Identifying  Dimensions/Taxa 
of  System  Performance 
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WORK  REQUIREMENTS 
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Figure  8.  Sample  Set  of  Attributes  Associated  with 
System  Work  Requirements 
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requirements);  insure  that  that  the  analysis  considers  all  factors  pertinent  to 
the  tasks;  insure  that  the  analysis  extends  completely  to  all  of  the  tasks 
selected  for  training;  maintain  quality  control  over  the  analysis  to  insure  that 
it  is  performed  accurately;  and,  ultimately,  see  to  it  that  the  analytic  results 
are  produced  when  needed,  and  fully  document  all  necessary  items  (e.g., 
domain  classifications,  skill  requirements,  etc.).  These  attributes  are  the 
features  of  the  system  that  the  designer  must  control  (Figure  1,  Block  3)  to 
see  to  it  that  the  task  analysis  work  requirement  is  satisfied. 

As  seen  in  the  APM  Overview,  these  are  also  the  very  same 
attributes  that  the  training  system  evaluator  must  measure  to  judge  whether 
that  task  analysis  work  requirement  is  satisfied  (Figure  1,  Block  4).  To 
appraise  the  system's  task  analytic  work,  the  evaluator  must  consider  whether 
appropriate  analytic  methods  were  selected;  whether  they  were  applied  to  serve 
the  intended  purposes,  and  whether  they  were  applied  in  manners  suited  to 
those  purposes;  whether  all  pertinent  factors  were  considered  in  the  analysis, 
and  whether  anv  extraneous  factors  were  included;  whether  all  tasks  selected 
for  training  were  analyzed,  and  whether  any  extraneous  or  superfluous  tasks 
were  analyzed;  whether  the  analyses  were  conducted  accurately;  and  whether 
the  intended  analytic  findings  were  produced. 

8 .  Design  and  Evaluation  as  an  Interactive  Process 


The  designer's  first  step  is  to  develop  a  preliminary  plan  for 
securing  the  identified  system  attributes.  The  plan  addresses  the  products  to 
be  delivered  and  the  processes  to  be  implemented,  but  its  principal  focus  is  on 
the  system's  required  potentialities.  The  designer's  main  task  at  this  stage  is 
to  demonstrate  that  the  system  will  have  the  prerequisite  abilities  needed  to 
carry  out  the  processes  and  deliver  the  products. 

In  parallel,  the  evaluator  examines  those  same  system  attributes  and 
develops  a  plan  for  testing  whether  the  system  has  the  attributes  and  satisfies 
the  work  requirements.  The  test  plan  includes  measures  and  measure 
application  procedures  for  all  attributes.  However,  at  this  stage  the  evaluator 
principally  is  interested  in  selecting  and  applying  measures  of  potential 
performance.  This  is  the  proposal  stage  of  design  and  evaluation.  The 
designer  says,  "Here  is  how  I  intend  to  see  to  it  that  the  system  will  have  the 
attributes  it  needs  to  work  properly."  The  evaluator  responds,  "Here  are  the 
specific  performance  defects  that  I  have  found  in  your  plan."  The  evaluator 
and  designer  work  together  to  determine  how  the  plan  should  be  modified  to 
correct  the  defects. 

Now  the  designer  implements  the  (modified)  plan.  The  focus  shifts 
from  potential  ("on  paper")  performance  to  in-process  performance,  as  the 
specified  activities  are  carried  out.  The  evaluator  again  applies  the  measures 
to  appraise  the  work,  but  now  interest  centers  on  the  measures  of  process 
attributes.  The  kinds  of  questions  that  arise  may  include:  Are  the  right 
kinds  of  activities  being  conducted?  Is  the  plan  being  followed?  Are  we  on 
schedule?  Are  procedures  being  adapted  properly  to  meet  the  needs  emerging 
in  the  real  world  of  system  development?  The  evaluator  monitors  the 
designer's  work,  and  from  time  to  time  ma;  report,  "What  you  are  doing  there 
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isn't  quite  what  is  needed."  In  a  cooperative  effort,  the  designer  and 
evaluator  modify  the  on-going  system  processes  to  correct  the  defects 
uncovered. 

At  some  point,  the  system  is  assembled  and  begins  to  deliver  its 
products  (Figure  1,  Block  7).  Again  the  evaluator  applies  the  measures.  The 
accent  now  is  on  whether  the  output  goods  and  services  satisfy  the  system's 
product  requirements.  If  defects  are  noted,  the  measures  of  product, 
process,  and  potentiality  help  to  diagnose  their  causes.  It  may  be  that  all 
that  is  needed  is  to  repeat  the  processes  more  carefully,  i.e.,  to  re-produce 
the  products  with  greater  attention  to  quality  control.  However,  the 
evaluation  might  disclose  that  it  is  the  processes  themselves,  or  even  their 
prerequisite  potentialities,  that  must  be  reconstructed.  An  evaluation  that 
interacts  cooperatively  with  the  design  effort  and  that  addresses  all  aspects  of 
performance  will  be  much  more  likely  to  help  develop  a  well  working  operational 
system  than  will  the  old-fashioned  "bottom  line  only"  approach  that 
traditionally  is  antagonistic  to  design. 

Design  and  evaluation  do  not  end  with  the  "christening"  of  the 
operational  system  (Figure  1,  Block  8).  They  remain  living  functions  as  long 
as  the  system  continues  to  operate,  always  looking  for  possible  improvements 
that  will  allow  the  system  to  respond  to  changing  needs  and/or  to  adopt 
newer,  better  technology  and  procedures. 

9.  Data  Bases  of  System  Performance  Taxa:  Focus  on  Training 

Applications 

Extensive  practical  application  of  the  APM  requires  the  availability  of 
taxonomies  of  system  performance.  Each  such  taxonomy  is  an  organized  data 
base.  Its  contents  are  the  work  requirements  of  the  particular  system,  or 
family  of  systems,  for  which  the  taxonomy  was  prepared.  A  design  or 
measurement  analyst  may  examine  the  total  data  base  and  identify  the  specific 
performance  elements  relevant  to  the  application  at  hand.  As  more  taxonomies 
are  developed,  the  scope  of  applications  available  to  the  model  will  increase. 

To  provide  one  such  data  base,  generic  taxonomies  were  developed 
for  major  functional  subsystems  of  training  systems.  The  decision  to  focus 
APM  development  on  training  applications  reflects  the  observations  that 
training  systems  are  pervasive,  critical  to  modern  military  operations  involving 
sophisticated  equipment,  and  consistent  with  ARI's  primary  interest  in  human 
behavioral  issues. 

Development  of  training  system  data  bases  for  APM  applications 
necessitated  identification  of  the  training  system  hierarchical  structure,  and 
subsequent  generation  of  performance  taxonomies  for  the  members  of  that 
structure.  It  is  functional,  not  organizational,  structure  that  is  of  interest 
here.  The  organizations  of  two  particular  training  systems  may  differ  widely, 
but  each  will  accomplish  the  same  basic  functions  required  of  any  training 
enterprise. 

The  principal  generic  function  of  training  systems  is  postulated  to  be 
learning ,  defined  as  any  activity  involving  the  senses  that  affects  behavior  in 
some  purposeful  fashion.  Learning  is  a  human  function:  it  is  people  who 
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carry  out  the  sensory  activities  to  experience  the  intended  behavioral  effects. 
The  principal  generic  operator/staff  member  of  a  training  system,  thus,  is  the 
learner. 

The  other  major  generic  function  of  training  is  postulated  to  be 
helping  to  learn,  defined  as  providing  an  efficient  learning  environment  to  the 
learner.  Numerous  types  of  people,  using  a  variety  of  equipment,  may  work 
in  any  given  training  system  to  make  it  conducive  for  the  learner  to  learn. 
All  such  people  are  learning  helpers.  They  acquire  more  specific  titles  in 
accordance  with  the  particular  types  of  help  they  provide. 

The  two  major  functions  can  be  broken  down  into  more  detailed 
operations,  or  subfunctions.  These  can  be  grouped  conveniently  into  six 
generic  training  subsystems,  as  illustrated  in  Figure  9. 

a.  The  administrative  control,  or  Command  Subsystem — which  deals 
with  identifying  needs  for  training,  allocating  resources, 
recruiting  training  personnel,  monitoring  and  evaluating  learner 
performance,  etc.  Command  Subsystem  is  staffed  principally  by 
training  administrators.  They  help  the  learner  to  learn  by 
constructing  and  managing  the  system  in  which  the  learning 
activities  can  take  place. 

b.  The  curriculum  development,  or  Design  Subsystem--which  is 
responsible  for  planning  the  instructional  activities,  selecting 
training  technology,  assembling  content  material,  defining 
instructor  and  student  requirements,  etc.  The  Design 
Subsystem  is  the  portion  of  the  training  system  in  which  the 
Instructional  Systems  Development  (ISD)  model  is  applied. 
Design  Subsystem  is  staffed  primarily  by  curriculum  developers. 
They  help  the  learner  to  learn  by  determining  the  specific 
behavioral  effects  that  are  needed  and  by  planning  the  sensory 
activities  that  will  lead  to  those  effects. 

c.  The  facilities  development,  or  Emplacement  Subsystem — which 
constructs  or  acquires  and  assembles  all  facilities  and  equipment 
needed  to  support  the  sensory  learning  activities.  The 
Emplacement  Subsystem  takes  the  plans  developed  by  the  Design 
Subsystem  and  insures  that  all  materials,  equipment, 
installations,  supplies,  etc.,  needed  to  carry  out  tho^e  plans 
are  made  available.  Emplacement  Subsystem  is  operated  mainly 
by  facilities  developers.  They  help  the  learner  to  learn  by 
providing  the  tools  needed  to  carry  out  the  learning  activities. 

d.  The  logistics  support,  or  Logistics  Subsystem — which  deals  with 
maintenance  of  facilities  and  equipment,  housing,  feeding  and 
recreation  of  training  system  personnel,  replenishment  of 
consumables,  transportation  of  people  and  supplies,  etc.  The 
logistics  supporters  help  the  learner  to  learn  by  attending  to 
the  myriad  of  details  necessary  to  keep  the  training  system 
running  smoothly  and  free  of  discomfort  and  distraction. 
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One  person  may  serve 
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more  than  one  subsys¬ 
tem  (i.e.,  an  individual 
may  "wear  two  or  more 
hats"  and  simultaneously 
serve  as  two  or  more 
different  kinds  of 
operator). 


Figure  9.  Training  Subsystems  and  Their  Operators 
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e.  The  instructor  preparation,  or  Enabling  Subsystem — which  deals 
with  familiarizing  instructors  with  the  plans,  content,  equipment 
and  facilities  involved  in  the  intended  learning  and  with 
preparing  the  instructors  to  teach  the  course  effectively.  The 
Enabling  Subsystem  also  is  responsible  for  tailoring  the  genera1 
plans  and  material  to  the  specific  learning  needs  of  a  particular 
class  or  team  of  learners.  The  principal  staff  of  the  Enabling 
Subsystem  are  the  instructors,  teachers,  professors,  etc.,  who 
directly  assist  the  learner  in  carrying  out  the  prescribed 
sensory  activities.  Collectively,  these  people  are  called 
facilitators.  They  help  the  learner  to  learn  by  presenting 
information ,  demonstrating  techniques,  coaching  the  learner's 
efforts,  evaluating  and  correcting  performance,  etc. 

f.  The  instructional  implementation,  or  Delivery  Subsystem — which 
is  where  the  learning/training  actually  takes  place.  In  the 
Delivery  Subsystem,  the  prescribed  sensory  activities  are 
carried  out,  and  the  learner  experiences  the  intended 
behavioral  effects.  The  principal  staff  member  of  the  Delivery 
Subsystem  is  the  learner,  aided  by  all  of  the  other  staff 
members  identified  above . 

Figure  10  presents  "what"  performance  is  required  of  each 
subsystem,  '.e.,  their  generic  objectives-level  taxa  of  potentialities,  processes 
and  products.  Those  taxonomies  were  compiled  by  applying  the  objectives- 
level  guidelines  presented  previously  in  Figure  7.  No  attempt  was  made  to 
identify  generic  taxa  of  environment  or  constraints,  since  the  scope  of  work 
did  not  allow  for  such  development. 

The  functional  purposes-level  guidelines  and  characteristics-level 
guidelines  of  Figure  7  were  applied  to  develop  complete  taxonomies  for  the 
Design,  Enabling  and  Delivery  Subsystems.  Those  are  presented, 
respectively,  in  Appendices  A,  B  and  C.  Those  taxonomies  provide  a  general 
data  base  for  addressing  virtually  any  design  or  evaluation  issue  concerning 
those  three  subsystems  in  virtually  any  training  system. 

10.  Sample  Derivation  of  Attributes  for  a  Training  System 

An  attribute  previously  was  defined  as  a  concrete,  observable 
feature  of  a  system  that  bears  directly  on  the  work  that  the  system  can 
perform.  For  each  taxon  of  performance  required  of  a  system,  there  exist 
certain  system  attributes  that  determine  whether  and  how  well  the  system  can 
satisfy  that  taxon.  The  designer  controls  those  attributes  to  make  the  system 
work  well.  The  evaluator  examines  those  attributes  to  assess  how  well  the 
system  works. 

To  aid  in  identifying  the  system  attributes  associated  with  a  given 
performance  taxon,  the  analyst  can  "decompose"  the  system  into  its  constituent 
parts.  The  "parts"  of  a  human-machine  system  are  the  people  who  operate  the 
system,  the  equipment/materials  employed  in  the  operation,  and  the 
procedures/methods  by  which  the  people  employ  the  equipment  and  interact 
with  each  other.  The  derivation  of  attributes  then  proceeds  as  follows: 
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Enabling 
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ing  management  resources 
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expertise 

•  Insuring  availability  of 
learning  guidance  expertise 
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tions 
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•  Analysis  of  facilitator's  ability  to  man¬ 
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of  the  curriculum 
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•  Qualified  facilitators 
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•  Curricula  tailored  to 
given  delivery  appli¬ 
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Emplacement 

.  Insuring  availability  of 
adequate  learning  facilities 
.  Insuring  availability  of 
adequate  learning  materials 
.  Insuring  availability  of 
adequate  instructional  aids 

•  Analysis  of  curriculum  to  determine 
emplacement  requirements 

.  Acquisition,  construction  and/or  modifi¬ 
cation  of  learning  facilities 

•  Acquisition,  construction  and/or  modifi¬ 
cation  of  learning  materia) 

•  Acquisition,  construction  and/or  modifi¬ 
cation  of  instructional  aids 

•  Appropriate  learning 
facilities  for  given 
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•  Insuring  availability  of 
adequate  personnel  support 

•  Insuring  availability  of 
adequate  facilities  mainte¬ 
nance 

•  Insuring  availability  of 
adequate  equipment  main¬ 
tenance 

•  Analysis  of  delivery  plans  to  determine 
logistics  requirements 

•  Development  of  schedule  of  personnel 
support  services 
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maintenance  services 

•  Development  of  procedures  for  restora¬ 
tive  maintenance 

•  Acquisition  of  facilities  and  materials 
needed  for  delivery  of  the  services 

.  Deliverance  of  the  services 

•  Personnel  support 
services 

•  Facilities  maintenance 
services 

•  Equipment  maintenance 
services 

Delivery 

.  Providing  learners  with  the 
learning  environment 

•  Insuring  learners'  interac¬ 
tion  with  the  environment 

•  Managing  learners'  progress 

•  Evaluating  learners' 
achievement 

•  Providing  bases  for  sys¬ 
tem  evaluation 

•  Analysis  of  specific  learner's  needs 

•  Conduct  preparation  learning  activities 

•  Conduct  presentatinn/demonstration 
learning  activities 

•  Conduct  application/practice 

•  Conduct  learner  testing 

•  Acquisition  of  formative  evaluation  data 

•  Appropriate  learning 
activities 

•  Learners  who  have 
achieved  the  stated 
objectives 

•  Assessment  of  learners 
achievements 

•  Formative  evaluation 
data 

Figure  10.  Objectives-Level  Performance  Taxa  of  Training  Subsystems 
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a.  Determine  which  of  the  system's  "parts"  are  involved  in  the 
performance  taxon  of  interest. 

Identify  the  specific  operators  or  classes  of  operators  who  are 
responsible  for  the  particular  potentiality,  process  or  product 
and/or  who  arc  included  in  that  potentiality,  process  or 
product.  Similarly,  identify  the  specific  equipment  or  materials 
used  in  or  contained  in  that  potentiality,  process  or  product. 
Finally,  determine  which  of  the  procedures/methods  by  which 
the  operators  employ  the  equipment  pertain  to  that  potentiality, 
process  or  product. 

b.  Identify  the  qualitative  and  quantitative  traits  or  properties 
that  the  involved  people,  equipment  and  procedures  must  have 
in  order  to  achieve  or  satisfy  the  performance  taxon  of 
interest. 

The  requisite  traits  might  include  specific  aptitudes,  expertise, 
strengths,  reliability,  fidelity,  etc.  These  traits  need  not  be 
directly  observable.  However,  they  must  be  directly  relevant 
to  the  particular  system  potentiality,  process  or  product  being 
studied.  And,  if  one  of  these  traits  is  not  observable,  it  must 
at  least  be  possible  to  infer  whether  or  not  the  trait  is  present. 

c.  Identify  concrete,  observable  features  of  the  involved  people, 
equipment  and  procedures  that  provide  bases  for  inferring 
reliably  whether  the  essential  qualitative  and  quantitative  traits 
are  present. 

These  concrete,  observable  features  are  the  attributes  of 
interest.  That  is,  they  are  the  controllable  and  measurable 
entities  that  determine  the  system's  achievement  of  the 
performance  taxon  of  interest. 

To  illustrate  this  process,  consider  Training  Design  Subsystem 
performance  taxon  1.2,  "Establishing  Performance  Objectives,"  (see  Appendix 
A).  Performance  objectives  can  be  defined  as  written,  explicit  descriptions  of 
specific  terminal  behaviors  that  learners  are  required  to  demonstrate  to  signify 
successful  training.  Each  performance  objective  states  something  specific  that 
the  learner  will  be  able  to  do  upon  completion  of  training.  (See  the  Glossary 
for  a  more  complete  definition  of  "Performance  Objective.")  Clearly,  the 
potential  for  establishing  performance  objectives  accurately  and  completely  is 
an  essential  ingredient  of  the  subsystem  responsible  for  developing  the 
training  curriculum. 

What  "parts"  of  the  Training  Design  Subsystem  are  involved  in  the 
potentiality  of  "Establishing  Performance  Objectives"?  First  and  foremost, 
specific  people  are  involved:  the  subsystem  must  designate  key  instructional 
system  designer  (s)  having  the  ability  to  establish  accurate,  complete 
performance  objectives.  Secondly,  important  materials  are  involved:  these 
materials  consist  of  information  or  data  that  disclose  what  it  is  that  the 
learners  need  to  be  able  to  do,  on  the  job,  following  the  end  of  their  training. 
Lastly,  instructional  design  methods  are  involved:  the  methods  of  interest  are 
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the  procedures  that  the  instructional  system  designer(s)  will  use  to  collect  and 
analyze  the  relevant  data,  and  to  translate  the  analytic  findings  into 
statements  of  performance  objectives.  In  order  to  demonstrate  that  their 
training  system  has  the  potential  for  "Establishing  Performance  Objectives," 
the  training  designers  must: 

°  Name  and/or  describe  the  individuals  who  will  establish  the 
objectives; 

°  Define  the  information  to  be  collected  and  analyzed  to  establish 
the  objectives; 

°  Define  precisely  how  the  information  will  be  collected  and 
analyzed. 

What  are  the  essential  qualitative  and  quantitative  traits  that  these 
people,  materials  and  methods  must  have  in  order  to  insure  the  potentiality  of 
"Establishing  Performance  Objectives"? 

Certainly,  the  people  designated  for  this  task  must  possess  expertise 
in  the  establishment  of  performance  objectives,  and  they  must  be  prepared  to 
devote  sufficient  level  of  effort  to  that  task  for  this  particular  training  design 
project.  The  information  to  be  used  as  the  basis  for  establishing  the 
objectives  must  be  appropriate  for  that  task,  and  it  must  be  accessible  to  the 
training  designers.  The  methods  to  be  used  to  collect  and  analyze  the 
information  likewise  must  be  appropriate,  and  adequate  resource  support  for 
implementing  those  methods  must  be  provided.  In  naming  their  staff  and  in 
defining  the  information  and  methods  that  will  be  used,  the  training  designers 
must  provide  concrete  evidence  that  these  essential  traits  will  be  present. 

What  is  the  concrete  evidence  that  the  traits  needed  for 
"Establishing  Performance  Objectives"  are  present?  That  is,  what  are  the 
system  attributes  that  can  be  directly  observed  and  examined  to  determine 
whether  the  training  system  satisfies  that  particular  taxon? 

Figure  11  lists  concrete,  observable  features  of  training  system 
people,  materials  and  methods  from  which  reliable  inferences  concerning  the 
system's  potential  for  "Establishing  Performance  Objectives"  could  be  drawn. 

1 1 .  Guidelines  for  Developing  Measures  Specifications 

A  measure  is  a  judgment  or  appraisal  about  the  thing-being- 
evaluated.  Thus,  a  measure  of  a  human-machine  system  attribute  is  an 
appraisal  of  the  "goodness"  of  that  attribute.  Each  measure  supplies  a 
particular  quantum  of  evidence  or  insight  about  the  degree  to  which  the 
system  possesses  some  feature  that  it  needs  to  perform  some  element  of  its 
work.  The  guidelines  for  systematizing  the  specification  of  measures  derive 
from  a  number  of  conceptual  bases,  or  postulates. 

°  A  system  attribute  is  a  variable  across  the  population  of 
human-machine  systems  tfTat  are  supposed  to  possess  the 
attribute. 
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Figure  11.  Sample  Attributes  Associated  with  a  Taxon  of 
Training  Design  Subsystem  Performance 
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The  value  that  the  attribute  takes  on,  in  any  given  system,  is 
a  function  of  the  degree  to  which  the  system  possesses  the 
attribute. 

The  attribute's  value  may  be  a  function  of  the  values  of 
simpler,  constituent  features  termed  the  dimensions  of  the 
attribute.  For  example,  if  "area"  is  a  relevant  attribute  of  a 
rectangle,  the  value  of  that  attribute  can  be  expressed  as  a 
function  (product)  of  the  rectangle's  base  and  height  values. 

In  some  cases,  it  may  be  possible  to  estimate  the  value  of  an 
attribute  directly.  In  other  cases,  it  may  be  possible  to 
estimate  the  values  of  the  attribute's  dimensions,  and  then  to 
compute  the  attribute  value. 

Different  kinds  of  estimates  of  an  attribute's  value  may  be 
made.  One  kind  is  an  estimate  of  absolute,  or  "true"  value 
(e.g.,  obtained  by  using  a  yardstick  to  determine  a  man's 
height).  A  second  kind  of  estimation  deals  with  relative  values 
of  an  attribute  (e.g.,  identifying  the  man  who  is  tallest,  2nd 
tallest,  3rd  tallest,  etc.).  Still  a  third  kind  of  estimate 
produces  threshold-referenced  attribute  values  (e.g.,  a  rod  of 
unknown  absolute  length  could  be  used  to  partition  the  group 
of  men  into  two  classes,  one  class  taller  and  the  other  shorter 
than  the  rod). 

A  measure  of  an  attribute  is  the  estimate  of  the  attribute's 
value.  Thus,  a  measure  may  be  direct  or  computed/ derived , 
may  be  absolute,  relative,  or  threshold-referenced,  and  also 
may  come  from  nominal,  ordinal  or  ratio  scales. 

Numerous  possible  measures  exist  for  any  given  attribute.  The 
evaluator's  principal  challenge  is  to  select  measures  that  provide 
the  kinds  of  estimates  of  attribute  values  that  are  appropriate 
to  the  measurement  purpose  at  hand,  and  that  permit  those 
estimates  to  be  made  accurately,  reliably,  and  efficiently. 

The  attribute  measures,  when  properly  combined,  will  supply 
appraisal  of  the  more  general  traits  with  which  the  attributes 
are  associated  (see  Section  10. b . ,  above) .  Similarly,  the  trait 
measures,  in  combination,  provide  the  judgment  of  the  system's 
performance  with  respect  to  the  particular  taxon  of  interest. 

The  taxon  measurement  requirements  thus  dictate  the  kinds  of 
estimates  that  are  needed  for  the  traits.  In  turn,  the  trait 
estimation  requirements  dictate  the  kinds  of  attribute  estimates 
that  are  needed. 

Figure  12  summarizes  these  concepts.  The  figure  represents  a 
general  hierarchy  of  measurement.  At  the  top,  to  which  all  branches  lead,  is 
the  desired  measurement  of  system  performance:  that  is  a  judgment  or 
appraisal  of  the  system's  satisfaction  of  a  particular  performance  taxon  of 
interest. 


Figure  12.  Schematic  Representation  of  a  General 
Hierarchy  of  Measurement 


At  the  second  level  appear  the  traits  or  properties  that  the  system 
must  possess  in  order  to  satisfy  the  taxon.  For  the  sake  of  generality,  n  of 
these  appear  in  Figure  12.  Each  trait  belongs  to  one  of  the  three  fundamental 
elements  of  a  human-machine  system:  the  people,  the  equipment,  or  the 
procedures.  The  measurement  of  system  performance  is  a  function,  f,  of  the 
individual  trait  measurements.  That  is  equivalent  to  saying  that  system 
performance  is  the  sum  total  of  the  contributions  of  the  system's  people, 
equipment  and  their  interactions.  By  measuring  at  the  trait  level,  the 
evaluation  process  permits  the  respective  contributions  of  these  fundamental 
system  elements  to  be  appraised,  and  functionally  combined  to  supply  the 
judgment  of  system  performance. 

But  any  given  trait  may  not  be  directly  observable,  and  hence  not 
directly  measurable.  Human  traits  in  particular,  but  also  equipment  and 
procedural  traits,  may  be  abstract.  However,  if  a  trait  is  real,  it  must  be 
possible  at  least  to  infer  it  from  concrete,  observable  factors.  Those  factors 
are  the  attributes  associated  with  the  trait.  The  measurement  of  any  given 
trait,  e.g.,  trait  No.  i,  is  a  function,  £i,  of  the  measurements  of  its 
individual  attributes.  Each  trait  has  its  own  set  of  attributes,  and  its  own 
functional  relationship  with  its  attributes. 

Every  attribute  is  concrete  and  observable.  Thus,  every  attribute 
is  at  least  theoretically  directly  measurable.  But  many  attributes  are  complex, 
and  difficult  or  expensive  to  measure  directly.  Often,  those  kinds  of 
attributes  are  multi-dimensional,  and  thus  their  measurements  may  be  computed 
from  the  values  of  simpler,  unidimensional  factors  that  are  more  amenable  to 
direct  measurement.  That  is,  the  measurement  of  a  particular  attribute,  e.g., 
attribute  No.  ij,  might  best  be  obtained  as  a  function,  hii,  of  the  values  of  its 
constituent  dimensions.  Each  dimension  is  a  factor  amenaole  to  practical  direct 
measurement. 

The  links  between  given  performance  taxa  and  their  related  attribute 
dimensions  may  vary  widely.  Consequently,  the  intervening  traits  and 
attributes  may  be  simpler  (even  identical  from  one  level  to  the  next)  or  more 
complex  as  well.  Furthermore,  the  number  of  levels  one  chooses  to  analyse  in 
the  measurement  hierarchy  of  Figure  12  may  also  be  a  function  of  what  is 
feasible,  how  diagnostic  one  wishes  to  be  and  what  is  cost-effective. 

The  concepts  discussed,  exemplified  and  depicted  above  provide  the 
basis  for  developing  guidelines  for  selecting  attribute  measures,  and  for 
specifying  how  those  measures  are  to  be  combined  to  supply  a  judgment  of  the 
system's  performance  relative  to  some  taxon  of  interest.  These  guidelines 
proceed  as  follows: 

a.  Determine  the  kind  of  performance  estimation  that  is  required 
and  appropriate  for  the  particular  taxon  and  measurement 
purpose  at  hand. 

b.  Determine  the  trait  value  estimates  that  are  needed  to  support 
the  required  taxon  measurement,  and  identify  the  functional 
relationship  between  the  trait  values  and  the  taxon  value. 
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c.  For  each  trait,  determine  the  attribute  value  estimates  needed 
to  support  the  trait  measurement,  and  identify  the  functional 
relationship  between  those  attribute  values  and  the  trait  value. 

d.  Identify  the  dimensions  of  each  attribute  that  may  be  used  to 
generate  the  required  value  estimates. 

e.  Identify  the  functional  relationship  between  the  attribute  value 
and  the  constituent  dimension  values. 

f.  Define  the  methods  by  which  the  dimension  values  will  be 
obtained. 

To  illustrate  the  application  of  the  measures  specification  guidelines, 
consider  the  following  hypothetical  situation: 

The  U.S.  /  rmy  Infantry  School  has  issued  a 
Request  for  Proposals  (RFP)  to  undertake 
development  of  a  training  system  to  support 
introduction  of  a  new  squad  automatic  weapon. 

Responses  have  been  received  from  several 
different  organizations.  A  committee  of  training 
development  experts  has  been  assembled  to  review 
the  proposals,  and  to  select  the  organization  best 
qualified  to  conduct  the  project.  The  committee 
has  6  members. 

For  simplicity,  this  illustration  will  address  only  the  following 
Measurement  Purpose: 

To  review  and  evaluate  a  set  of  proposals  received  from 
training  development  organizations  to  identify  the  organization 
best  suited  to  establish  performance  object; .  es  for  squad 
automatic  weapon  training. 

The  guidelines  (a-f)  are  applied  as  follows: 

a.  Determine  the  kind  of  performance  estimation  required  for  this 
measurement  purpose. 

A  relative  appraisal  of  the  competing  organizations'  capabilities 
for  establishing  training  objectives  will  suffice.  The  estimates 
should  permit  the  organizations  to  be  rank-ordered  (e.g.,  "best 
qualified,"  "2nd  best,"  ...  etc.). 

b.  Determine  the  trait  value  estimates  needed  to  support  this 
performance  measurement.  Identify  the  functional  relationship 
between  the  trait  values  and  the  performance  measurement. 

The  traits  essential  for  satisfying  the  potentiality  of 
establishing  training  objectives  are  (from  Figure  10,  earlier): 


1)  Expertise  in  establishing  performance  objectives 

2)  Adequate  level  of  effort  proposed 

3)  Appropriateness  of  information  proposed  for  collection 

4)  Accessibility  of  the  proposed  information 

5)  Appropriateness  of  data  collection/analysis  procedures 

6)  Sufficiency  of  proposed  resources 

Every  member  of  the  review  committee  will  review  each 
proposal,  and  will  make  an  independent  judgment  of  each 
proposal  with  respect  to  the  above-listed  traits.  Each  trait  will 
be  assigned  a  value  from  0  (insufficient)  to  3  (excellent). 

Thus,  each  reviewer  will  assign  each  proposal  a  value  from  0 
(totally  insufficient  on  all  traits)  to  18  (excellent  relative  to  all 
six  traits).  In  turn,  each  proposal  will  receive  a  summary 
score  from  0  (totally  deficient  in  all  respects  to  all  reviewers) 
to  108  (excellent  in  all  respects  to  all  reviewers).  Proposals 
will  be  rank-ordered  in  accordance  with  their  respective 
summary  scores. 

c.  For  each  trait,  determine  the  attribute  value  estimates  needed 
to  support  the  trait  measurement.  Identify  the  functional 
relationship  between  the  attribute  values  and  the  trait 
measurement.  For  example: 

Trait  No.  4:  Accessibility  of  proposed  information 

Each  offerer's  proposal  will  be  assessed  relative  to  the 
following  two  attributes  to  arrive  at  a  measurement  of  this 
trait: 

1)  Evidence  that  the  offerer  has  secured  or  can  secure 
permission  to  extract  the  proposed  data  from  the 
proposed  source(s). 

2)  Evidence  that  appropriate  data  of  sufficient  quantity 
and  quality  exist  at  the  offerer's  proposed  source(s). 

Each  reviewer  will  make  an  independent  assessment  of  these 
attributes  for  each  proposal.  Relative  to  attribute  1),  the 
reviewer  will  assign  a  value  from  0  (cannot  be  done)  to  2  (can 
be  done) . 

Relative  to  attribute  2),  the  reviewer  will  assign  a  value  from  C 
(not  in  existence)  to~3  (exists  in  high  quality  and  quantity). 

After  the  reviewer  has  assessed  a  proposal  relative  to  attributes 
1)  and  22,  he/she  will  compute  the  measurement  of  Trait  No.  4 
using  the  table  set  forth  below. 
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Value  of  attribute  2) 

I — 0 - 1 - 2 - 3" 

Value  of 
attribute  1) 


o s o o 
1  0  0  12 

2  0  12  3 


Table  entries  are  values  for  the  "Accessibility  of  Proposed 
Information"  (Trait  No.  4).  Other  traits  are  determined  using 
similar  procedures  defined  by  the  analyst. 


d.  For  each  attribute,  identify  the  dimensions  that  may  be  used  to 
generate  the  required  value  estimates.  For  example: 


Attributes  associated  with  Trait  No.  4:  Accessibility  of 
proposed  information 

1)  Dimensions  of  the  attribute:  "Evidence  that  offerer 
has  secured  or  can  secure  permission  to  extract  the 
proposed  data  from  the  proposed  source(s)." 

Each  reviewer  will  consider  the  following  dimensions 
in  assessing  the  evidence  of  the  offerer's  ability  to 
access  the  proposed  source (s): 

a)  The  explicit  documentation,  in  the  offerer's 
proposal,  that  access  to  the  proposed  source(s) 
has  been  granted  to  this  offerer  for  this  project 
by  authoritative  representative  (s)  of  the 
source(s) . 

b)  Documentation,  in  the  offerer's  proposal,  of 

his/her  prior  success  in  accessing  similar 

information,  for  other  projects,  from  the  same 
proposed  source(s). 

c)  Documentation,  in  the  offerer's  proposal,  of 

his/her  prior  success  in  accessing  similar 

information  from  source (s)  similar  to  those 
proposed. 

d)  Potential  impediments  to  access  of  the  proposed 

source (s)  that  the  offerer  has  identified  in 
his/her  proposal,  but  has  not  adequately 

addressed . 

e)  Potential  impediments  to  access  of  the  proposed 
source(s)  known  to  the  reviewer  but  not 
identified  or  addressed  in  the  offerer's  proposal. 

N.B.  The  reviewer  shall  immediately  inform  the 
chairman  of  the  review  committee  of  any  such 
impediments. 
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f)  Reviewer's  knowledge  of  the  prior  history  of  the 
proposed  source(s)  in  granting/refusing  access 
to  this  or  similar  offerer. 

N.B.  The  reviewer  shall  immediately  convey  any 
such  knowledge  to  the  chairman. 

The  reviewer  will  consider  no  dimension  or  factor 
other  than  the  six  listed  above  when  appraising  this 
attribute,  unless  directed  to  do  so  by  the  review 
committee  chairman. 

2)  Dimensions  of  the  attribute:  "Evidence  that 

appropriate  data  of  sufficient  quantity  and  quality 
exist  at  the  offerer's  proposed  source(s)." 

(not  included  in  this  sample) 

e.  For  each  attribute,  identify  the  functional  relationship  between 
the  attribute  value  and  the  constituent  dimension  values.  For 
example: 

1)  Attributes  associated  with  Trait  No.  4:  Accessibility  of 
proposed  information 

a)  Functional  computation  of  the  value  of  the  attribute 
"evidence  that  offerer  has  secured  or  can  secure 
permission  to  extract  the  proposed  data  from  the 
proposed  source (s)." 

The  reviewer  must  assign  this  attribute  a  value  of  2 
(highest  possible  value)  if: 

0  Explicit  documentation  is  included  in  the 
offerer's  proposal  to  establish  that 
grant-of-access  explicitly  has  been  received  from 
authoritative  representative(s)  of  all  proposed 
sources,  or 

0  All  of  the  proposed  sources  are  known  to  grant 
access  to  all  users  similar  to  this  offerer,  for 
purposes  similar  to  those  intended  in  the 
proposed  project. 

The  reviewer  must  assign  this  attribute  a  value  of  0 
if: 

0  Any  key  source  (i.e.,  the  only  source  proposed 
by  this  offerer  for  accessing  certain  essential 
data)  is  known  to  refuse  access  to  all  users 
similar  to  this  offerer,  for  purposes  "similar  to 
those  intended  in  the  proposed  project,  or 
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Any  key  source  is  known  to  have  refused  access 
to  this  offerer  in  the  past,  for  purposes  similar 
to  those  proposed,  and  the  offerer  has  not 
demonstrated  a  method  for  securing  future 
access  in  his/her  proposal. 

In  the  absence  of  all  of  the  four  conditions  listed 
above,  the  reviewer  will  exercise  his  or  her  own  best 
judgment  in  assigning  a  value  of  0,  1,  or  2  to  this 
attribute.  The  reviewer  is  to  be  prepared  to  explain 
the  rationale  for  his  or  her  judgment. 

b)  Functional  computation  of  the  value  of  the  attribute 
"evidence  that  appropriate  data  of  sufficient  quantity 
and  quality  exist  at  the  offerer's  proposed 
source (s) . " 

(not  included  in  this  sample) 

f.  Define  the  methods  by  which  the  dimension  values  will  be 
obtained . 

The  principal  method  by  which  reviewers  will  obtain  values  for 
the  dimensions  described  under  item  d.,  above,  will  be  through 
a  careful  and  complete  study  of  the  oTferer's  proposal. 

In  addition,  for  certain  dimensions,  a  reviewer  may  possess 
relevant  prior  information  that  might  affect  the  values  of  those 
dimensions.  When  a  reviewer  believes  that  he  or  she  possesses 
such  information,  that  information  should  be  conveyed  to  the 
review  committee  chairman  as  soon  as  possible.  The. chairman 
will  decide  whether  the  information  is  to  be  considered  in 
assessing  the  value  of  a  particular  dimension.  Upon  so 
deciding,  the  information  will  be  conveyed  to  other  reviewers. 

Occasionally,  due  to  the  availability  of  relevant  prior 
information,  the  chairman  may  determine  that  a  particular 
dimension  must  be  assigned  a  particular  value  by  all  reviewers. 

12.  Initial  Concepts  for  Developing  Design  Specifications 

Although  the  project's  scope  did  not  allow  for  as  much  development 
in  the  APM's  design  specification  application  as  for  the  performance 
measurement  applications,  a  preliminary  set  of  concepts  has  been  prepared. 
They  are  summarized  briefly  in  this  section. 

A  design  specification  of  a  human-machine  system  is  a  description  of 
exactly  what  the  system  design  should  include  in  order  for  the  system  to 
achieve  all  of  its  performance  requirements.  As  with  the  APM's  measures 
specification  procedures,  the  notion  of  a  systematic  design  specification  process 
rests  on  a  number  of  conceptual  bases,  or  postulates: 
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c  Design  and  measurement  deal  with  precisely  the  same  issues: 
namely,  the  concrete,  observable  attributes  that  make  the 
system  what  it  is  and  that  cause  it  to  work  in  the  way  that  it 
does. 

°  The  design  specifier,  like  the  evaluator,  thus  must  know  how 
the  concrete,  controllable  attributes  are  linked  to  the  taxa  they 
serve.  Specifically,  the  design  specifier  must  know  the 
functional  relationship  between  a  taxon  and  its  essential  traits, 
and  the  functional  relationship  between  each  of  those  traits  and 
its  constituent  attributes.  Only  then  can  he/she  identify  the 
attribute  design  choices  to  make  to  effect  the  appropriate  status 
of  the  taxon. 

°  The  design  specifier's  choices  determine  the  value  that  each 
attribute  will  have  in  the  system  being  built.  TFose  choices 
determine  the  system's  trait  values  through  the  functional 
relationships  that  link  the  traits  and  the  attributes.  Similarly, 
the  trait  values  determine  the  values  of  the  performance  taxa. 

0  The  design  specifier's  intention  is  to  make  the  set  of  choices 
that  maximizes  the  values  of  the  taxa.  Often,  but  not  always, 
that  involves  attempting  to  maximize  the  attribute  values.  In 
some  cases,  the  best  choice  may  be  to  arrange  for  certain 
attributes  to  take  on  sub-maximum  values,  because  the  resulting 
combination  produces  the  best  attainable  trait  value. 

°  In  sum,  in  making  choices  the  design  specifier  needs  to  know 
and  to  apply  the  functional  relationships  that  link  the  system 
attributes  to  the  traits,  and  also  the  functional  relationships 
that  link  the  traits  to  the  taxa. 

0  The  design  specifier  also  needs  to  know  how  to  assess  the 
values  of  the  attributes,  traits  and  taxa,  to  insure  that 
appropriate  design  choices  are  made.  That  is,  the  design 
specifier  needs  to  know  the  measures. 

The  concepts  described  above  provide  the  basis  for  defining  an 
initial  set  of  guidelines  fer  developing  design  specifications.  That  effort  is 
left  for  future  projects. 

B .  Applications  of  the  APM  for  Measurement 

The  APM  concepts  and  guidelines  for  measurement  applications  are  next 
illustrated  with  more  realistic  examples.  These  illustrations  summarize  more 
complete  descriptions  to  be  found  in  the  earlier  project  reports  (Bloom,  et  al., 
1981,  1982).  Two  narrowly  focused  measurement  illustrations,  related  to  the 
Bradley  Infantry  Fighting  Vehicle  (BIFV),  are  given:  (!)  Derivation  of  a 
BIFV  Measures  Hierarchy  for  the  Surveillance  Function,  and  (2)  Derivation  of 
Measurable  Performance  Objectives  for  the  BIFV  Training  System. 
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1 .  Derivation  of  a  BIFV  Measures  Hierarchy  for  the  Surveillance 
Function 


a.  General  Conditions 


This  example  begins  with  a  general  look  at  many  BIFV 
performance  aspects,  of  which  surveillance  is  only  one.  The  application  to 
surveillance  is  pursued  after  establishing  that  more  general  context.  For  the 
general  case,  the  overall  measurement  purpose  was  taken  to  be: 

c  To  evaluate  the  combat  readiness  of  a  BIFV  squad  serving 
as  the  gounding  element  of  a  platoon  in  bounding 
overwatch  during  an  offensive  maneuver  within  an  armored 
threat  environment. 

This  measurement  purpose  was  chosen  for  several  reasons: 

0  First,  the  BIFV  weapons  system  is  a  complex  system  and 
any  attempt  to  apply  the  APM  to  a  unit  size  larger  than  a 
squad  would  require  analyses  beyond  the  scope  of  the 
present  effort. 

0  Second,  a  mounted  operation  with  the  squad  activity 
limited  to  conducting  bounding  overwatch  maneuvers  and 
an  engagement  with  a  stationary  armored  threat  were 
selected  primarily  to  keep  the  number  of  variables 
manageable. 

0  Third,  it  was  intended  that  this  application  would  serve  as 
a  useful  experience  in  utilizing  available  system 
documentation  and  could  adapt  information/doctrine  from 
similar  type  systems  as  would  a  military  analyst  in 
applying  the  methodology  to  a  new  system. 

Certain  assumptions  made  were  with  regard  to  the  qualifications 
of  the  BIFV  squad/platoon  personnel  and  the  BIFV  Table  of  Organization  and 
Equipment  (TOE).  These  assumptions  were  as  follows: 

0  The  BIFV/ Squad/ Platoon  personnel  are  MOS-qualified  and 
have  completed  the  necessary  unit  team  training  in  order 
to  participate  in  combat  readiness  exercises. 

0  The  BIFV  squad  is  manned  by  TOE-authorized  levels.  For 
the  purpose  of  this  application,  the  assumotion  was  made 
that  the  BI^V  squad  consists  of  nine  personnel,  i.e., 
Track  Commander,  Gunner,  Driver  and  six  (6)  Firing  Port 
Operators,  one  of  whom  is  a  fire  team  leader. 

0  The  BIFV  is  prepared  for  this  mission  with  maximum 
authorized  TOE  supplies  and  equipment. 
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The  B1FV  squad  will  remain  mounted  during  the  entire 
tactical  operation  and  would  only  dismount  personnel  if 
absolutely  necessary,  e.g.,  prepare  vehicle  for  fording, 
etc. 

b.  Determine  the  Hierarchical  Structure  in  Which  the  System  of 
Interest  is  Imbedded 


Figure  6  (presented  earlier)  depicts  the  external  view  of  the 
hierarchical  structure  providing  the  immediate  context  of  the  BIFV  Squad.  For 
this  measurement  purpose,  the  system  of  interest  is  imbedded  in  a 
subsystem/ system/ suprasystem  structure  that  encompasses  a  BIFV  Platoon. 
Within  the  system  of  interest  there  are  two  subsystems:  (1)  the  Carrier  Team 
Subsystem,  consisting  of  the  Track  Commander,  Gunner  and  Driver,  with  their 
associated  work  stations  and  equipment,  and  (2)  the  Crew  Compartment  Team 
Subsystem,  consisting  of  the  six  Firing  Port  Weapons  Operators  and  their 
associated  work  stations  and  equipment.  The  two  subsystems  could  be  further 
divided  into  sub-subsystems  which  would  bring  the  level  to  individual  system 
operators  and  their  associated  equipment.  However,  considering  the 
measurement  purpose,  this  level  is  beyond  the  application  interest  of  the  APM. 
The  collateral  systems  include  two  (2)  other  BIFV  squads,  one  (1)  BIFV 
squad/platoon  headquarters  and  the  threat  system(s).  The  suprasystem  is  the 
BIFV  platoon. 


In  applying  the  APM,  one  should  develop  the  total  number  of 
taxonomic  sets  to  which  the  system  of  interest  belongs,  including  sets  for  each 
subsystem  and  corresponding  sets  for  the  collateral  systems  and  for  the 
suprasy stems.  The  development  of  taxonomies  for  the  various  system  levels 
leads  to  the  identification  of  system-level  interactions  which  influence  relevant 
performance  requirements  and,  in  turn,  affect  performance  measurement  issues 
for  ihe  system  of  interest. 

c.  Define  and  Review  Scenarios  and/or  Documents  Required  for 

This  Application 

In  order  to  become  thoroughly  familiar  with  the  BIFV 
Squad/Platoon  system,  available  system  documents  were  reviewed.  Those 
documents  included  the  BIFV  Materiel  Needs  document,  BIFV  Systems 
Specifications,  Mission  Scenarios,  BIFV  System  Task  Descriptions,  and  Field 
Manuals  (for  similar  systems)  that  provided  guidance  with  regard  to  tactical 
operations.  As  a  point  of  departure  and  initial  guidance  with  regard  to  the 
measurement  purpose,  use  was  made  of  the  Mission  Scenario  #1  (Bounding 
Overwatch  Operation)  from  the  BIFV/CFV  Personnel  Selection  Study  (Bloom  et 
al.,  1979).  The  first  page  only  of  that  scenario  is  reproduced  here  in 
Figure  13.  A  squad  verbal  order  was  also  developed  to  serve  as  guidance, 
and  to  aid  in  the  identification  of  system  population  categories  relevant  to  the 
contextual  stages  of  the  measurement  process.  The  verbal  order  is  presented 
in  Figure  14.  Its  associated  map  is  not  shown  here. 

Specifying  the  measurement  purpose  also  requires  description  of 
the  measurement  conditions.  These  include  such  factors  as  the  geographic 
area(s)  in  which  the  system  is  to  perform  its  mission,  the  time  period  during 
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Figure  13.  Mission  Scenario  (First  page  only  for  sample  purposes) 
Mission  Segment:  Offensive  Operation:  Squad  is  Bounding  Element 
Bounding  Overwatch  -  Contact  Expected 


BITUAT10N 

"We  arc  going  to  conduct  on  attack  at  0600  hours  today.  This  terrain 
Sketch  IVe  made  represents  the  (round  we've  got  to  cover.  At  you  can  set 
the  attack  will  cover  a  distance  of  about  3  kilometers.  These  little  hilts. 

Ml  and  646  are  platoon  intermediate  objectives.  Hill  563,  west  of  Lahm  I? 
the  platoon  abjective.  We  are  presently  on  Hill  STS.  From  what  I  was  ab  r 
to  ate  from  the  limited  reeon  with  the  platoon  leader,  it  looks  like  the  enemy 
has  withdrawn  to  positions  north  of  Lahm.  Although  I  didn't  see  any  armored 
vehicles,  LLDavies  said  the  enemy  has  been  identified  as  part  of  a  motonted 
rifle  unit.  That  means  we  have  to  be  on  the  lookout  for  BMPs  and  maybe  some 
tanks.  We  may  also  encounter  such  obstacles  as  minefields  and  wide  trrrrhes. 
The  minefields  are  more  than  likely  going  to  be  forward  of  the  trenchlinr 

"Throughout  the  attack  we  will  stay  mounted.  The  1st  Platoon  will  be  on 
our  left  and  the  3rd  Platoon  on  our  right.  The  platoon  objective  is  going  to  be 
hit  with  artillery  and  mortar  fire  just  before  we  go  in.  The  company  has  tanks 
attached  to  aupport  us  through  the  attack. 

MISSION 

"Our  mission  is  to  attack  each  of  these  intermediate  objectives  and  final 
objective.  We  have  to  clear  the  trenches  and  bunkers  in  our  tone.  Upon  secur¬ 
ing  Hill  553  we  are  to  prepare  defensive  positions.  Once  we're  there  and  see  the 
terrain  we'll  talk  more  about  it.  We  will  move  out  at  0600." 

EXECUTION 


"When  we  move  out,  the  platoon  is  going  to  be  using  bounding  overwatch 
and  we  are  the  bounding  squad.  With  the  platoon  overwatching  us  from  this  hill, 
we  will  move  up  this  draw  until  we  get  to  the  patch  of  woods.  1  couldn't  see 
im-ch  of  these  woods;  so  once  we  get  close  I'll  guide  the  track  ovn  the  remainder 
ui  the  route.  As  soon  as  the  rest  of  the  platoon  moves  up  to  us,  we'll  bound 
around  to  the  left  and  up  this  ridge  line.  I  could  see  it  pretty  clearly,  and  I 
don't  think  anything  is  up  there.  When  the  rest  of  the  platoon  has  moved  up 
and  all  the  tracks  art  in  position  we'U  wait  for  Lt.  Davies’  order  to  bound  to 
the  first  intermediate  objective.  Hill  549.  Once  we've  taken  Hill  549  we'll  wait 
for  Lt.  Davies'  order  to  continue  bounding  overwatch.  It's  important  that  we 
don't  get  loo  far  ahead  or  behind  the  rest  of  the  company.  If  we  detect  a 
mine  field  we'll  try  to  go  around  it  so  keep  your  eyes  open." 

"If  we  have  to  dismount,  the  platoon  sergeant  will  be  controlling  the  fires 
of  the  carrier  team  but  that  is  very  unlikely.  Our  priority  is  to  engage  any 
armored  vehicles  in  our  rector  and  then  bunkers  " 

"When  we  get  to  the  trenches  and  enemy  contact  la  made,  Sgt.  Jonas 
will  control  the  fires  of  the  FPWO's.  Don't  take  any  chances;  lay  down  a 
good  base  of  fire.  Report  all  pooaiblc  targets  for  the  carrier  team." 

RERV1CE  SUPPORT 

*lgt  Jonaa.  you  make  sure  that  the  squad  compartmtnt  and  erew  have 
the  required  basic  load  and  aquipment.  Lot  me  know  when  you're  all  complete. 
I'll  check  the  carrier  team  supplies  and  equipment." 

"McCarthy,  the  POL  track  will  ba  hare  around  0430;  maka  sure  our  fuel 
tank  la  topped  off." 

"Take  eare  of  any  casualties  within  the  trseki  give  him  some  quick  first 
aid  and  get  back  to  your  position.  The  company  aid  track  will  come  forward 
just  as  aeon  as  oach  intermediate  and  final  objective  is  secured." 

COMMAND  AND  SIGNAL 

"We  will  cross  the  SP  at  0600  on  Lt  Davits'  hand  signal.  Wc  will  main¬ 
tain  radio  silence  until  we  reach  Hill  549.  During  the  attack  no  unnecessary 
Intercom  chatter." 

"In  case  I  get  hit.  Igt.KowalWci,  you  taka  ovtri  our  call  sign  la  TANGO  11." 

"OK,  the  time  la  now  0230;  I  will  inspect  at  0515  hours.  It  will  be  daylight 
at  0545.  Wa  have  leas  than  4  hours  befora  wc  move  out.  We  will  maintain  radio 
silence  unto  contact  with  the  enemy  la  made.  The  p>atocn  and  the  squad  will 
tse  arm  and  hand  signals  during  the  maneuvers.  Remember  to  maintain  light 
and  noi-o  dieeipline.  Do  you  have  any  questions?  If  not,  get  ready." 


Figure  14.  Verbal  Squad  Order 

Purpose:  To  Attack  Hills  549,  546  and  553  (Lichtenfels,  Deutschland) 


39 


which  the  system  is  to  perform  its  mission,  the  weather  conditions  and  any 
other  limiting  variables.  As  noted  in  the  verbal  squad  order  for  this 
application,  the  geographic  area  selected  is  a  temperate  zone  of  operations  in 
Southern  Germany;  the  tim#1  period  of  performance  is  0600  to  1200  hours;  the 
weather  is  defined  as  fair,  no  precipitation,  and  good  visibility. 

d.  Begin  Applying  the  APM  to  the  System  of  Interest  to  Derive 

'taxonomy  Sets 

The  analyses  of  the  two  subsystems  were  initiated  first,  since  it 
was  felt  that  combining  the  subsystem  taxonomies  would  simplify  the  derivation 
of  the  taxonomy  for  the  system  of  interest  itself.  The  initial  subsystem 
selected  for  analysis  was  the  Carrier  Team.  The  APM  revision  was  applied  to 
Carrier  Team  and  Crew  Compartment  Team  subsystems,  separately,  to  derive 
the  fifteen  different  taxonomy  subsets.  These  correspond  to  the 
"intersections"  of  each  of  the  three  levels  of  system  description  with  each  of 
the  five  contextual  stages  of  the  measurement  process.  Figure  5  (presented 
earlier)  is  a  sample  of  that  effort,  and  incorporates  the  decimal  number 
indexing  scheme  described  earlier. 

At  the  Objectives  Level,  the  population  categories  were  assigned 
successive  numbers  in  the  first  decimal  position.  For  example,  at  the 
Objectives  Level  under  Potentialities,  each  population  category  was  identified 
as  1.1,  1.2,  1.3,  etc.  To  be  specific,  the  taxa  identified  on  the  Objectives 
Level  system  description  for  the  Potentialities  of  the  BIFV  Carrier  Team  were 
designated  as  follows: 

1 . 1  Providing  surveillance 

1.2  Conducting  weapons  fire 

1.3  Providing  squad  command  and  control 

1.4  Providing  transportation 

1.5  Providing  communication 

1.6  Providing  squad  protection 

The  Functional  Purposes  Level  descriptions  were  assigned  a 
two-decimal  code,  i.e.,  1.1.1,  1.1.2,  1.1.3,  etc.  To  continue  the  above 
example,  the  Functional  Purposes  Level  taxa  of  "1.1  Providing  surveillance" 
were  designated  as: 

1.1.1  Conducting  sector  surveillance 

1.1.2  Detecting  targets 

1.1.3  Acquiring  targets 

1.1.4  Identifying/classifying  targets 

1.1.5  Tracking  targets 

The  Characteristics  Level  of  Potentialities  was  assigned  a 
three-decimal  code,  i.e.,  1.1. 1.1,  1.1. 1.2,  1.1. 1.3,  etc.  In  subdividing  the 
functional  purpose  "1.1.1  Conducting  sector  surveillance,"  the  characteristics 
which  describe  what  capabilities  the  system  must  have  in  order  to  satisfy  that 
application  were  numbered  as  follows: 

1.1. 1.1  Unaided  visual  sector  surveillance  capability 

1.1. 1.2  Aided  visual  sector  surveillance  capbility 

1.1. 1.3  Infrared  sensor  sector  surveillance  capability 
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e .  Derivator)  of  a  BIFV  Measures  Hierarchy  for  the  Surveillance 

Function 

As  a  test  of  the  utility  of  the  resulting  BIFV  taxonomy,  a  set 
of  measures  was  derived  from  those  taxa  associated  with  the  Carrier  Team 
Unit's  performance  of  surveillance.  The  advantages  in  choosing  the 
performance  element  of  surveillance  were  that,  first,  it  is  a  requirement 
common  to  many  military  human-machine  systems  and,  second,  it  entails  many 
potentialities,  processes,  and  products  of  performance  and  is  subject  to  the 
influence  of  many  environmental  and  constraining  factors.  Thus,  as  a  test  of 
the  measures-derivation  process,  it  permitted  a  wide  variety  of  system  taxa  to 
be  included  and  offered  the  possibility  of  producing  measures  relevant  to  many 
system  applications. 

Figure  15  displays  the  Carrier  Team  System  taxa  associated  with 
the  "surveillance"  performance  element.  On  the  Objectives  Level  of  system 
descript:on,  these  taxa  include: 

°  The  basic  capability  (potentiality)  of  surveillance. 

0  The  process  of  sector  surveillance,  for  both  air  and 
ground  sectors. 

°  The  product  of  threat  detections  (for  which  surveillance  is 
a  prerequisite). 

°  The  physical  and  climatological  environment. 

°  Surveillance  constraints. 

°  Threat  constraints  (the  nature  of  the  threats,  and  the 
tactical  response  required,  may  affect  how  the  system 
performs  surveillance). 

All  taxa  that  "descend"  from  these  on  the  Functional  Purposes  Level  and  the 
Characteristics  Level  also  are  included. 

The  numbers  shown  in  parentheses  in  Figure  15  refer  to  the 
taxonomy  codification  scheme  illustrated  earlier  in  Figure  5.  They  allow  the 
links  among  the  three  levels  of  description  to  be  indicated  explicitly.  For 
example,  taxon  1.1. 2.1  "descends"  from  taxon  1.1.2,  which  in  turn  "descends" 
from  1.1. 


The  taxa  arrayed  in  Figure  15  could  next  be  used  as  search 
terms  for  accessing  the  body  of  published  literature  on  human-machine  systems 
measurement ,  with  a  view  toward  identifying  measures,  analytic  methods,  and 
other  test  procedures  that  previously  were  applied  successfully  to  evaluate 
similar  systems.  For  purposes  of  this  example,  however,  it  was  not  deemed 
necessary  to  conduct  a  formal  literature  review.  Instead,  the  staff  members 
relied  on  their  own  system  measurement  experience  with  surveillance  systems 
(especially  those  in  the  infantry  and  air  defense  artillery  contexts)  as  the 
source  from  which  measures  relevant  to  the  listed  taxa  would  be  drawn. 
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Figure  15.  Carrier  Team  System  Taxa  Associated  with 
"Surveillance”  Performance  Element 


Figures  16,  17,  and  18,  respectfully,  depict  the  measures  that 
are  potentiality-oriented,  process-oriented,  and  product-oriented.  The  first 
set  of  measures  (Figure  16)  are  intended  to  answer  questions  about  the  basic 
capabilities  for  surveillance  that  the  system  possessed.  The  second  set 
(Figure  17)  answers  questions  concerning  how  the  system  performs  its  various 
surveillance  activities.  The  third  set  (Figure  18)  addressed  questions 
concerning  the  results  of  those  surveillance  activities. 

Each  set  of  measures  is  organized  in  conformance  to  the  APM's 
three  levels  of  system  description.  The  uppermost  row  in  each  figure  contains 
only  those  measures  associated  with  Objectives  Level  taxa.  The  second  row  is 
reserved  for  measures  associated  with  Functional  Purposes  Level  taxa.  All 
lower  rows  deal  with  measures  associated  with  Characteristics  Level  taxa. 
Measures  arc  labelled  with  reference  to  the  particular  attribute  (taxon)  of 
performance  on  which  they  bear.  Each  such  taxon  is  itself  designated  by  a 
numeric-decimal  label.  Each  measure  associated  with  a  given  taxon  is  labelled 
by  that  taxon's  numeric-decimal,  with  parenthetic  alphabetic  postscript.  For 
example,  three  measures  are  identified  in  Figure  16  for  the  "Surveying  Sector" 
taxon  designated  by  1.1.1  (a  Functional  Purposes  Level  taxon  under  the 
Performance  Potentialities  column).  Those  three  measures  are  designated  by 
1.1.1(A),  1.1.1(B),  and  1.1.1(C),  respectively. 

One  important  point  that  bears  re-emphasis  is  that  all  measures 
listed  in  F'gures  16,  17  and  18  focus  on  the  system's  performance  of 
surveillance.  Each  measure  derives  from  some  particular  system  taxon,  but 
the  measure  is  concerned  only  with  the  interaction  of  that  taxon  with 
surveillance.  If  some  other  aspects  of  system  performance  (e.g.,  "Squad 
Command  and  Control,"  "Communications")  had  been  under  study,  the  relevant 
taxa  might  have  included  some  items  that  are  similar  to  those  listed  for 
"surveillance. "  However,  even  if  the  taxa  were  identical  for  the  .several 
performance  aspects,  different  measures  would  most  likely  have  been  derived 
because  of  differing  measurement  purposes. 

Some  examples  can  clarify  the  preceding  point.  One  of  the 
product-oriented  taxa  associated  with  surveillance  is  "Tactical  Decisions"  (Item 
5.2.3,  on  the  Functional  Purposes  Level  in  Figure  IS).  The  association  stems 
from  the  fact  that  surveillance  is  a  principal  source  of  input  for  BIFV  Squad 
tactical  decisions.  If  an  analyst  wishes  to  undertake  a  comprehensive 
assessment  of  the  system's  performance  of  surveillance,  one  issue  that  must  be 
addressed,  therefore,  is  "How  well  does  the  system's  surveillance  provide 
input  to  its  tactical  decisions?"  It  is  precisely  that  issue  that  is  addressed  by 
the  measures  listed  under  "Tactical  Decisions"  in  Figure  1R.  The  reader  will 
note  that  those  measures  have  nothing  directly  to  do  with  the  kinds  of  tactical 
decisions  made,  with  the  correctness  of  those  decisions,  with  the 
decision-making  process  itself.  The  measures  are  concerned  only  with  the 
kinds  and  quantities  of  tactical  decision  input  data  supplied  or  overlooked  by 
the  surveillance  process.  Those  measures  are  applied  in  the  context  of  tactical 
decisions,  but  they  remain  focused  on  surveillance. 

Suppose  for  the  moment,  however,  that  measures  were  being 
sought  not  for  the  "Surveillance"  performance  element,  but  rather  for  "Squad 
Command  and  Control."  Tactical  decisions  would  also  be  included  in  the  set  of 
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Figure  16.  BIFV  Surveillance:  Potentiality-Oriented  Measures 


Figure  17.  BIFV  Surveillance:  Process-Oriented  Measures 


Threat  Detections 
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Figure  18.  BIFV  Surveillance:  Product-Oriented  Measures 


taxa  associated  with  that  performance  element  because  the  formulation  of 
tactical  decisions  is  an  integral  part  of  the  exercise  of  command.  Now, 
however,  the  key  question  would  be,  "How  well  does  the  system  formulate  its 
tactical  decisions,  given  the  information  available?"  The  measures  needed  to 
address  that  question  would  focus  on  the  kinds  of  decisions  made,  the  decision 
algorithms  used,  the  volumes  of  decisions  made,  their  correctness,  their 
timeliness,  etc.  In  short,  different  sets  of  measures  might  derive  from  a 
common  system  taxon,  depending  upon  the  performance  element  of  interest. 
The  basic  point  is  that  the  taxa  are  abstractions  of  measures.  The  concrete 
specification  of  the  measures  they  represent  depend  on  exactly  which  facet  or 
element  of  the  system's  performance  is  being  studied. 

The  measures  hierarchy  depicted  in  Figures  16,  17  and  18,  of 
course,  is  an  idealized  set.  That  is,  no  attempt  has  been  made  to  determine 
the  feasibility  of  using  any  of  the  measures,  or  to  eliminate  those  that  may 
require  data  that  cannot  be  obtained  under  realistic  circumstances.  That 
"weeding  out"  activity  properly  belongs  at  a  later  stage  in  the  overall 
measurement  process.  The  purpose  of  the  taxa  is  to  guide  the  analyst  in 
identifying  and  organizing  a  set  of  measures  that,  if  they  can  be  used,  would 
answer  all  the  research  questions  at  hand.  The  taxa  themselves  shed  little  or 
no  light  on  the  practicality  of  those  measures. 

There  is  no  conclusive  test  or  set  of  standards  that  can  be 
used  to  judge  the  merits  of  the  "Surveillance"  measures  hierarchy.  The  true 
merits  of  any  measure  can  be  known  only  when  the  measure  is  actually 
employed  under  the  intended  circumstances.  Nevertheless,  the  measures 
themselves  appear  intuitively  to  be  well  suited  to  assess  the  BIFV  Squad's 
performance  of  surveillance.  The  measures  would  allow  the  analyst  to  proceed 
in  a  careful  and  logical  step-by-step  fashion.  Applied  in  the  proper  sequence, 
they  would  address  the  following  kinds  of  questions: 

0  What  capabilities  does  the  BIFV  Squad  possess  in  the  area 
of  surveillance? 

0  What  surveillance  capabilities  did  the  system  actually  use 
in  this  particular  measurement  application? 

0  How  were  these  capabilities  affected  by  the  environmental 
and  other  circumstances  that  pertained  to  this  particular 
measurement  application? 

°  What  procedures  did  the  system  actually  use  to  apply  its 
surveillance  capabilities? 

0  How  did  these  procedures  differ  from  the  prescribed 
surveillance  procedures? 

0  How  did  the  environmental  and  other  circumstances  affect 
the  surveillance  procedures? 

For  what  specific  applications  were  the  surveillance 
procedures  conducted? 
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0  Was  sufficient  and  appropriate  information  or  other  input 
supplied  to  those  applications  through  surveillance? 

0  What  specific  deficiencies  were  found  in  the  information/ 
input  provided  by  surveillance? 

Achieving  the  ability  to  answer  questions  such  as  these  in 
every  analytic  application  would  constitute  substantial  progress  in  the 
state-of-art  of  human-machine  systems  measurement.  In  this  specific  instance 
at  least,  the  APM  has  produced  a  measures  set  that  apparently  is  structured 
around  the  right  kinds  of  research  questions,  and  that  possesses  the  logical 
sequential  organization  required  to  address  those  questions. 

2.  Derivation  of  Measurable  Performance  Objectives  for  the  BIFV 
Training  System  (BIPVTS) 

a.  General  Conditions 


In  this  second  sample  illustration  of  applying  the  APM  for 
Measurement,  the  focus  is  shifted  to  the  training  system  associated  with  the 
BIFV.  The  information  in  these  paragraphs  summarizes  a  more  extensive  and 
detailed  discussion  reported  in  an  earlier  project  document  (Bloom,  et  al. , 
1982,  Section  IIC  and  Appendix  A).  Using  the  training  system  hierarchical 
structure  defined  earlier  in  this  report,  and  illustrated  in  Figure  9,  this  APM 
application  was  conducted  for  the  purpose  of  deriving  measures  to  assess  how 
well  the  Design  subsystem  of  the  BIFVTS  specifies  the  testing  of  gunner 
trainees. 


b .  Determine  the  Hierarchical  Structure  in  Which  the  System  of 
Interest  Is  Imbedded 


Figure  19  shows  the  system  hierarchy  for  the  BIFVTS — that  is, 
its  relationship  to  other  training  systems.  Every  training  system  can  be 
conceived  of  as  depicted  previously  in  Figure  9,  with  six  subsystems 
(Command,  Design,  Enabling,  Emplacement,  Logistics  and  Delivery). 

The  BIFVTS  Command  subsystem  is  operated  and  equipped  by 
the  BIFV  Task  Force,  an  ad  hoc  subsystem  of  the  U.S.  Army  Infantry  School. 
The  BIFV  Task  Force  has  been  created  and  staffed  to  exercise  overall 
responsibility  for  the  design,  development,  testing,  and  final  production  and 
dissemination  of  BIFV  training.  The  specific  immediate  responsibilities  of  the 
Task  Force  are  to: 

Identify  the  specific  skills  and  knowledge  required  of  BIFV 
operators . 

0  Develop  BIFV  training  strategies. 

0  Establish  new  equipment  training  (NET)  for  the  BIFV. 

0  Administer,  monitor,  and  evaluate  all  BIFV  training 
development,  testing,  and  production. 
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Figure  19.  Major  Systems  Interacting  with  BIFVTS 
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The  reader  will  recognize  that  several  of  the  above-listed 
responsibilities  are  actually  training  design  functions.  Indeed,  the  BIFVTS 
Design  subsystem  also  is  operated  (at  present)  solely  by  members  of  the  BIFV 
Task  force.  Their  current  design  work  is  oriented  toward  the  production  of 
five  distinct  curricula: 

0  An  "add-on"  course  for  Skill  Level  1  BIFV  Soldiers  (MOS 

11M10). 

0  A  Basic  Gunnery  Course  (for  Skill  Level  2  Soldiers  [MOS 
11M20]  and  some  Skill  Level  1  Soldiers). 

A  BIFV  Commander's  Course  (for  Skill  Levels  3-5  [11M30, 
11M40,  11M50]  and  for  officer  grades  01-05). 

°  A  transitional  new  equipment  training  (NET)  course 
covering  the  above-listed  skill  levels. 

°  A  Master  Gunner's  Course. 

The  Master  Gunner's  curriculum  principally  is  intended  as  an 
input  to  the  BIFVTS  Enabling  subsystem.  One  E7  or  E8  from  the  S3  section  of 
each  BIFV  mechanized  infantry  battalion  will  receive  Master  Gunnery  training, 
as  will  one  E6  from  each  BIFV  company.  The  Master  Gunner's  duties  include 
the  administration,  supervision  and  delivery  of  Basic  Gunnery  training  within 
his  or  her  unit.  The  Master  Gunner,  in  effect,  will'  be  a  field 
facilitator/administrator  within  BIFVTS.  Other  current  or  planned  operations 
within  the  BIFVTS  Enabling  subsystem  include  the  organization  and  deployment 
of  a  team  of  facilitators  to  support  new  equipment  training  on  a  transitional 
basis  as  the  BIFV  is  introduced  into  operational  units;  familiarizing  .selected 
mechanized  infantry  instructor/facilitators  with  the  emerging  BIFV  curricula; 
and  delivering  Mastery  Gunnery  training.  The  Enabling  subsystem  must  have 
the  potentiality  for  insuring  that  adequate  instructional  skills  resources  are 
provided  for  BIFVTS  Delivery  applications. 

The  BIFVTS  Emplacement  subsystem  is  responsible  for  the 
design  and  selection/fabrication  of  all  audiovisuals  and  other  learning  aids 
recessary  to  support  implementation  of  the  various  curricula,  and  for  the 
acquisition  or  construction  and  preparation  of  all  sites  at  which  learning 
activities  will  take  place,  whether  in  a  classroom  or  field  setting.  One 
important  element  of  this  responsibility  will  be  the  design  and  construction  of 
gunnery  practice  ranges.  The  range  requirements  presently  envisioned 
include: 

°  Firing  port  weapons  range 

°  BIFV  Basic  Gunners  range 

0  BIFV  Squad  subcaliber  range 

0  BIFV  Squad  qualification  range 

°  Platoon  qualification  attack  range 

0  Platoon  qualification  defense  range 

Note  that  all  of  the  BIFVTS  Emplacement  subsystem's 
performance  requirements  derive  from  the  need  to  insure  that  all  facilities  and 
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materials  necessary  to  support  delivery  of  DIFV  training  are  available  and 
operable  at  the  times  and  places  needed. 

The  BIFVTS  Logistics  subsystem  presently  is  operated  solely  as 
a  component  of  the  Fort  Benning  logistic  support  system.  In  time, 
increasingly  greater  shares  of  the  operation  of  that  subsystem  will  be 
exercised  by  various  mechanized  infantry  units  in  the  field. 

The  BIFVTS  Delivery  subsystem  will  allow  and  insure  that 
learning  takes  place  in  the  contexts  of  the  various  curricula  produced  by  the 
Design  subsystem.  Soldiers  and  officers  will  carry  out  prescribed  sensory 
activities,  guided  and  assisted  by  facilitators,  and  will  achieve  the  specified 
performance  objectives  applicable  to  their  BIFV  operational  assignments. 

BIFVTS  thus  is  ?  well  structured  training  system,  possessing 
functionally  oriented  subsystems  compatible  with  the  general  training  model  and 
pursuing  a  broad  range  of  administrative,  developmental,  and  delivery  goals  to 
support  the  introduction  of  the  new  BIFV  into  the  mechanized  infantry.  It 
does  not  detract  at  all  from  that  image  to  note  that  BIFVTS  can  also  be  viewed 
as  a  loose  consortium  of  smaller  scale,  relatively  independent  training  systems 
that  pursue  more  narrowly  defined  goals.  These  smaller  systems  are  organized 
around  the  five  BIFVTS  curricula  now  in  development.  These  systems  may  be 
denoted  as: 

°  The  BIFV  NET  System 

0  The  Basic  BIFV  Soldier  Training  System 

0  The  Basic  BIFV  Gunnery  Training  System 

°  The  Master  BIFV  Gunnery  Training  System 

°  The  BIFV  Commander  Training  System 

In  order  to  reduce  the  scope  of  this  trial  application  to  a 
practical  level,  the  focus  is  narrowed  to  the  Basic  BIFV  Gunnery  Training 
System.  Like  all  systems,  this  one  operates  in  the  context  of  an  interactive 
system  network  of  suprasystems ,  collateral  systems,  and  subsystems.  The 
network  is  shown  in  Figure  20.  The  Basic  Gunnery  system  is  subservient  to 
such  entities  as  the  U.S.  Army  Infantry  School,  the  Fort  Benning  Logistic 
Support  System  and,  of  course,  the  total  BIFVTS.  It  interacts  collaterally 
with  the  other  four  independent  BIFV  training  systems  and  with  portions  of 
U.S.  Army  Infantry  units,  namely,  those  portions  from  which  the  E4's  and 
E5's  will  be  recruited  as  candidate  gunners.  It  also  interacts  collaterally  with 
the  BIFV  Carrier  Team  subsystem.  That  is  the  portion  of  the  total  BIFV 
system  for  which  the  gunner-learners  are  to  become  qualified  as  operators. 
And  finally,  the  Basic  BIFV  Gunnery  Training  System  has  a  fraternal 
(collateral)  twinship  with  the  Basic  BCFV  Gunnery  Training  System.  The 
subsystems  of  Basic  BIFV  Gunnery  Training  are,  of  course,  the  six  familiar 
subsystems  of  Command,  Enabling,  Design,  Emplacement.  Logistics,  and 
Delivery . 


The  focus  of  analysis  was  narrowed  even  further  by  addressing 
only  the  Design  Subsystem  of  BIFV  Basic  Gunnery  Training.  Figure  21 
depicts  the  network  of  systems  that  interact  most  closely  with  the  Design 
subsystem  of  Basic  BIFV  Gunnery  Training.  As  in  any  training  system, 
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Figure  20.  Major  Systems  Interacting  with  the 
Basic  BIFV  Gunnery  Training  System 
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(Continued) 
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System  of  Interest:  The  Design  Subsystem  cf  the  Basic  BIFV  Gunnery  Training  System 

(Continued  — ► ) 


Figure  21.  Sample  Hierarchical  Structure  Centered  on 
a  Subsystem  of  a  Training  System 
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Figure  21.  Sample  Hierarchical  Structure  Centered  on 
a  Subsystem  of  a  Training  System  (Concluded) 
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Gunnery  Training  Design  is  subservient  to  the  Command  subsystem,  and 
collateral  to  the  Enabling,  Emplacement,  Logistics,  and  Delivery  subsystems. 
Key  sub-subsystems  of  both  Emplacement  and  Delivery  have  been  included  in 
Figure  21  to  emphasize  the  special  gunnery  instruction  and  practice 
requirements  that  must  be  reflected  in  Design's  products.  Gunnery  Training 
Design  is  also  collateral  to  the  Design  subsystems  of  the  other  BIFV  Training 
"packages."  This  is  because  all  of  those  packages  share  some  performance 
objectives,  learning  activities,  etc.,  with  gunnery  training. 

The  sub-subsystems  of  Gunnery  Training  Design  correspond  to 
the  thus  far  identified  segments  of  the  gunnery  curriculum.  These  represent 
the  sequential  stages  of  learning  through  which  BIFV  gunner-learners  must 
pass  to  become  fully  qualified.  The  initial  stages  stress  individual  training; 
the  latter,  team  training. 

c.  More  About  the  Measurement  Application:  Specification  of 
Gunner-Learner  Testing 


Having  selected  the  Design  subsystem  of  Basic  BIFV  Gunnery 
Training  as  the  system  of  interest  for  this  trial,  it  remains  to  select  an  issue 
of  focus  for  the  application  (i.e.,  an  assessment  of  how  well  Design  meets  some 
limited  aspect  of  its  performance  requirements).  Ultimately,  all  assessments  of 
training  system  or  subsystem  performance  must  supply  some  portion  of  the 
answer  to  this  fundamental  question:  Did  the  learners  learn  what  they  were 
supposed  to  learn? 

Design's  basic  job  is  to  provide  training  specifications.  It 
specifies  what  is  to  be  learned,  the  context  of  learning,  and  the  learning 
activities.  One  sub-issue  of  specifying  learning  activities  was  selected  for  this 
trial  applic  ..ion:  specifying  gunner-learner  testing.  The  other  sub.-issues  of 
that  heading  are:  specifying  gunner-learner  preparation  activities;  specifying 
activities  for  making  presentations  (or  demonstrations)  to  gunner-learners;  and 
specifying  gunner-learner  application  (or  practice)  activities.  Virtually  all 
learning  theorist:-  agree  that  preparation,  presentation,  application,  and 
evaluation  (or  testing)  constitute  the  four  classes  of  learning  activities.  The 
issue  of  how  well  a  Design  subsystem  specifies  learner  testing  materials  and 
procedures  is  of  special  importance  in  any  training  system.  No  training 
system  can  afford  to  allow  its  Design  subsystem  to  produce  poor  measures  of 
learners'  achievements.  The  Basic  BIFV  Gunnery  Training  System  is  no 
exception . 

d .  Begin  Applying  the  APM  to  the  System  of  Interest  to  Derive 

Performance  Attributes  and  Measures 

Before  moving  on  to  the  next  step  in  this  trial  application,  in 
which  particular  performance  taxa  associated  with  learner  testing  specification 
will  be  identified,  it  is  worthwhile  to  set  the  stage  by  Informally  reviewing 
some  of  the  requirements  and  implications  oi  learner  testing  in  any  training 
application.  The  following,  at  least,  seem  almost  self-evident: 

°  Testing  must  be  objectives-referenced 

The  purpose  of  testing  is  fo  evaluate  learners'  progress 
toward  or  achievement  of  the  performance  objectives,  i.e., 
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that  which  is  to  be  learned.  No  performance  objective 
should  go  untested.  No  test  that  does  not  relate  to  some 
performance  objective  should  be  included. 

Testing  must  be  criteria-referenced 

It  is  not  enough  that  a  test  simply  requires  the  learner  to 
exercise  the  behavior  called  for  in  a  performance  objective. 
The  test  must  disclose  whether  the  behavior  is  performed 
">ell  enough"  to  satisfy  learning  needs. 

Some  kind  of  pre-training  testing  is  necessary 

All  training  assumes  some  prerequisite  behavior.  If 
nothing  else,  it  is  always  assumed  that  the  learner  is  able 
to  carry  out  the  sensory  activities  called  for  in  the 
training.  Many  training  applications  admit  the  possibility 
that  some  learners  may  already  have  achieved  certain  of 
the  specified  performance  objectives.  There  is  no  point  in 
commencing  training  unless  the  learner  first  is  tested  to 
insure  that  he  or  she  possesses  the  assumed  prerequisites. 
A  very  informal  test  might  suffice  for  that  purpose, 
perhaps  only  a  records  check  or  even  a  cursory  visual 
examination  of  the  learner,  but  some  testing  for 
prerequisites  surely  is  needed.  When  there  is  reason  to 
believe  that  some  learners  may  already  have  achieved  the 
learning  objectives,  pre-testing  of  those  objectives  also  will 
be  necessary. 

Some  testing  for  feedback  is  necessary 

Testing  is,  after  all,  a  learning  activity.  Learners  need 
to  know  how  well  they  are  doing  in  order  to  progress  and 
persevere.  Any  training  system  should  provide  at  least 
some  testing  for  the  purpose  of  feeding  back  progress  and 
diagnostic  information  to  the  learners. 

Testing  must  diagnose  learning  deficiencies 

It  is  not  sufficient  that  tests  disclose  that  a  learner  cannot 
perform  the  intended  behavior.  In  order  to  provide 
meaningful  feedback  to  the  learner  and  to  facilitate 
subsequent  improvement  of  the  curricul;  .n  and  the 
delivery,  tests  that  diagnose  the  specific  ingredients  of 
substandard  learner  performance  must  be  included  in  the 
specifications.  Only  then  can  the  learner's  performance  be 
corrected  and  brought  up  to  standard. 

Testing  must  be  reflective  of  on-the-job  performance 

The  tests  specified  must  require  the  learners  to  manifest 
the  job-relevant  behaviors  under  job-relevant  conditions. 
The  testing  conditions  should  resemble  job  conditions  as 
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closely  as  possible,  since  the  test  outcome  otherwise  may 
not  be  a  valid  predictor  of  the  learner's  job  performance. 

0  Some  kind  of  qualification  testing  is  necessary 

No  learner  should  be  "certified"  as  having  successfully 
completed  training  unless  he  or  she  demonstrates,  through 
testing,  that  the  performance  objectives  have  been 
achieved.  Depending  on  the  circumstances  of  the 
particular  training  experience,  the  qualification  testing 
might  be  formal  or  informal,  but  some  testing  surely  is 
needed. 

The  preceding  list  probably  would  occur  to  any  evaluator 
interested  in  measuring  the  adequacy  of  a  Design  subsystem's  specification  of 
its  learner  testing  program.  With  that  list  in  mind,  the  next  step  is  to  review 
the  Design  subsystem  performance  taxa  of  Appendix  A  for  relevance  to 
measuring  the  adequacy  of  specifying  the  gunner-learner  testing  program. 

Two  hundred  and  eighty-one  taxa  of  Design  Subsystem 
performance  requirements  are  listed  in  Appendix  A.  These  are  distributed 
among  the  three  aspects  of  performance  (potentialities,  processes,  and 
products)  and  across  the  three  levels  of  system  description  (objectives, 
functional  purposes,  and  characteristics).  Some  of  the  281  taxa  are  relevant 
to  evaluation  of  a  Design  Subsystem's  performance  in  specifying  learner 
testing.  Others  are  not.  At  the  present  stage  of  development  of  the  APM, 
well  defined  procedures  for  determining  which  taxa  do  or  do  not  bear  on  a 
given  measurement  application  are  yet  to  be  established.  Indeed,  one  purpose 
of  this  trial  application  was  to  add  to  the  base  of  experience  from  which  such 
procedures  may  be  derived.  Currently,  assessment  of  taxa  relevance  is  a 
"judgment  call"  by  an  analyst  or  team  of  analysts  working  together.  The 
thought  process  can  be  described  as  follows:  "Here  we  have  some  aspect  of 
this  sytem's  performance,  and  over  here  we  have  a  performance  taxon  for  the 
system.  That  taxon  represents  a  particular  requirement  that  the  system  is 
supposed  to  satisfy.  Suppose  the  system  does  a  bad  job  in  trying  to  satisfy 
that  requirement.  Might  that  have  any  bearing  on  the  particular  aspect  that 
we  are  studying?"  If  the  analyst  concludes  that  the  answer  is  "yes,"  the 
taxon  in  question  would  be  included  among  the  subset  from  which  measurement 
issues  and  implications,  and  ultimately  measures,  will  be  derived.  If  the 
answer  is  "no,"  the  particular  taxon  will  supply  no  input  to  the  measurement 
application  at  hand. 

Identification  of  relevant  taxa,  then,  is  a  step-by-step  march 
through  an  entire  system  taxonomy,  during  which  each  taxon  is  picked  up  in 
turn,  examined,  and  finally  accepted  or  rejected  in  accordance  with  whether  or 
not  it  appears  to  the  analyst  to  have  something  of  interest  to  say.  This 
process  can  be  streamlined  somewhat,  simply  by  examining  only  the 
characteristics-level  taxa.  If  a  taxon  on  the  characteristics  level  is  found  to 
be  relevant  to  the  particular  measurement  application,  then  the  functional 
purposes-level  taxon  from  which  it  derives  also  must  be  relevant.  A 
characteristics-level  taxon,  after  all,  is  simply  a  detailed  performance 
requirement  that  the  system  must  satisfy  if  it  is  to  achieve  the  related 
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functional  purpose.  Thus,  the  functional  purpose  must  be  subsumed  under 
the  aspect  of  performance  being  studied.  Similarly,  if  a  functional 
purposes-level  taxon  is  relevant  to  the  measurement  application,  so  must  be 
the  objectives-level  taxon  from  which  it  derives.  The  functional  purpose, 
after  all,  is  only  a  required  application  of  the  basic  potentiality,  process,  or 
product  defined  on  the  objectives  level.  For  example,  if  the  analyst  in  using 
Appendix  A  were  to  conclude  that  taxon  number  2. 1.3.1  (a  characteristics- 
level  process  requirement)  is  of  significance  to  the  measurement  application,  he 
or  she  would  automatically  know  that  taxa  2.1.3  and  2.1  (its  functional 
purposes-level  and  objectives-level  antecedents)  also  are  relevant. 

An  examination  was  made  (Bloom,  et  al.  ,  1982)  of  every 
characteristics-level  taxon  of  potentiality  of  the  general  Training  Design 
subsystem  for  possible  relevc  'e  to  assessment  of  the  subsystem's  performance 
in  specifying  learner  testing.  Rationale  for  concluding  whether  an  individual 
taxon  is  nr  is  not  relevant  to  the  measurement  issue  is  presented  there. 
Similar  (but  undocumented)  examinations  were  made  of  the  process  and  product 
taxa.  The  final  conclusion  of  these  examinations  was  that  119  of  the  281  taxa 
are  relevant  to  evaluation  of  the  Design  Subsystem's  specifications  of  learner 
testing.  Those  relevant  taxa  form  14  taxonomic  subsets,  of  which  5  are 
potentialities,  3  are  processes,  and  6  are  products.  Figure  22  is  a  sample  of 
4  of  the  14  subsets. 

Alongside  each  of  the  evaluative  issues  arrayed  in  Figure  22, 
the  reader  will  find  a  set  of  attributes  deemed  to  be  relevant  to  the  particular 
issue.  Each  attribute  is  a  concrete  element  of  the  Design  subsystem's  work. 

It  is  something  that  can  be  seen,  examined,  submitted  to  analysis,  taken 
apart,  etc.  In  short,  each  attribute  is  something  that  is  measurable. 
Further,  each  attribute  is  something  that,  when  measured,  can  supply  a 
portion  of  the  answer  to  the  question  posed  in  the  evaluative  issue. 

An  example  may  help  to  clarify  the  relationship  between  issues 
and  attributes.  Consider  the  fifth  issue  in  Subset  No.  6  (Figure  22),  "Does 
the  system  accurately  identify  the  physical  skills  that  BIFV  gunners  must 
acquire  to  perform  their  tasks,  so  that  appropriate  tests  of  physical  skills  may 
be  specified  and  developed?"  This  issue  arises  from  a  characteristics-level 
process,  i.e.,  the  analytic  process  used  to  identify  the  physical  skills  learners 
must  acquire  in  order  to  serve  as  BIFV  gunners.  Clearly,  tests  of  all 
required  physical  skills  must  be  specified  and  administered  as  part  of  the 
gunner  certification  process.  If  the  system's  analysis  is  deficient,  so  that 
certain  essential  physical  skills  are  not  identified,  or  so  that  some  irrelevant 
physical  skills  are  mistakenly  identified  as  essential,  the  testing  specifications 
also  will  be  deficient.  As  the  design  work  progresses,  the  sponsors  certainly 
would  wish  to  know  whether  a  thorough  and  accurate  analysis  of  required 
physical  skills  is  underway.  In  order  to  assess  that  issue,  the  evaluator 
would  look  first  at  the  analytic  methods  the  training  designers  are  using  to 
identify  required  physical  skills.  Are  those  methods  appropriate  to  the 
analytic  needs  at  hand?  Are  all  of  the  data  required  for  applicaton  of  those 
methods  being  obtained  and  used  appropriately?  Is  there  any  fault  to  find 
with  the  way  in  which  the  training  designers  are  applying  those  methods?  As 
the  analyses  begin  to  produce  results,  the  evaluator  would  examine  the 
physical  skills  requirements  identified.  Are  all  of  those  skills  really  pertinent 
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PART  1:  Issues  and  Attributes  of  "Potentialities"  (Subsets  1-5) 


Subset  No.  2;  'Establishing  Performance  Objectives" _ 

Issues  Attributes 


1.2  Does  the  system  have  the  ability 

to  insure  that  all  BIFV  gunnery 
performance  objectives  will  be 
identified  so  that  they  may  be 
addressed  in  gunner  testing  spe¬ 
cifications? 


1.2.1  Can  the  system  insure  that  all 
things  BIFV  gunners  will  be  able 
to  do  after  training  will  be 
stated,  so  that  tests  of  all 
required  abilities  may  be 
specified? 

1.2.1. 1  Can  the  system  insure  that  the 
stated  abilities  will  include 
definitions  of  the  required 
gunner  actions,  so  that  the  test 
specifications  may  address  those 
actions? 

1.2. 1.2  Can  the  system  insure  that  the 
stated  abilities  will  include 
definitions  of  the  appropriate 
conditions  under  which  BIFV  gun 
ners  are  to  perform  the  actions, 
so  that  the  test  specifications 
may  reflect  those  conditions? 

1.2. 1.3  Can  the  system  insure  that  the 
stated  abilities  will  include 
definitions  of  the  appropriate 
standards  of  acceptable  perfor¬ 
mance  of  the  BIFV  gunner 
actions,  so  that  the  test  speci¬ 
fications  may  incorporate  those 
standards? 

1.2.2  Can  the  system  insure  that  the 
identified  BIFV  gunnery  perfor¬ 
mance  objectives  will  be  applied 
to  provide  a  basis  for  objec¬ 
tively  assessing  gunners'  per¬ 
formances? 

1.2.2. 1  Can  the  system  insure  that 
appropriate  pretest  items 
addressing  the  performance  ob¬ 
jectives  will  be  developed,  so 
that  they  may  be  included  in  the 
testing  specifications? 

1.2. 2. 2  Can  the  system  insure  that 
appropriate  posttest  items 
addressing  the  performance  ob¬ 
jectives  will  be  developed,  so 
that  they  may  be  included  in 
the  testing  specifications? 


0  Scope  of  information  available  con¬ 
cerning  BIFV  gunners'  tasks 
0  Staff  experience  and  qualifications 
in  establishing  performance  objectives 
0  The  planned  approach  to  establishing 
BIFV  gunnery  performance  objectives 
0  The  resources  allocated  to  establish¬ 
ing  BIFV  gunnery  performance  objec¬ 
tives 

0  The  elements  or  factors  of  gunner 
abilities  intended  to  be  stated 
0  Intended  formats  for  stating  BIFV 
gunner  abilities 

0  The  plans  for  stating  intended 
abilities 

0  The  types  of  performance  actions 
intended  to  be  defined 
0  The  plans  for  defining  actions 
required  of  BIFV  gunners 


0  The  elements  or  factors  of  perfor¬ 
mance  condition  intended  to  be 
defined 

°  The  plans  for  defining  BIFV  gunnery 
performance  conditions 


0  Elements  or  factors  of  performance 
standards  intended  to  be  defined 
°  The  plans  for  defining  standards  for 
BIFV  gunnery  performance  actions 


0  Elements  of  the  basis  for  assessment 
intended  to  be  provided 
0  Plans  for  providing  a  basis  for  objec¬ 
tively  assessing  BIFV  gunners' 
performances 


0  Types  of  BIFV  gunnery  pretest  items 
intended  to  be  developed 
0  Plans  for  developing  BIFV  gunnery 
pretest  items 


0  Types  of  BIFV  gunnery  posttest  items 
intended  to  be  developed 
0  Plans  for  developing  BIFV  gunnery 
posttest  items 


Figure  22.  Measurement  Issues  and  Measurable  Attributes  Relevant  to 
Evaluation  of  BIFV  Gunnery  Testing  Specifications 


-59- 


PART  2:  Issues  and  Attributes  of  "Processes"  (Subsets  6-8) 


Subset  No.  6:  "Analyze  Tasks  Selected  for  Training" 

Issues  Attributes 


2.3  Does  the  system  conduct  a  proper 

analysis  of  BIFV  gunner  tasks, 
so  that  appropriate  tests  of 
those  tasks  may  be  specified? 


2.3.2  Does  the  system  accurately  iden¬ 
tify  the  domains  of  learning 
Involved  in  the  BIFV  gunner 
tasks,  so  that  tests  may  be  spe¬ 
cified  and  developed  in  the 
proper  domains? 


2. 3. 2.1  Does  the  system  accurately  iden¬ 
tify  the  knowledge  that  BIFV 
gunners  must  acqvire  to  perform 
their  tasks,  so  that  appropriate 
tests  of  knowledge  may  be  speci¬ 
fied  and  developed? 

2. 3. 2. 2  Does  the  system  accurately  iden¬ 
tify  the  mental  skills  that  BIFV 
gunners  must  acquire  to  perform 
their  tasks,  so  that  appropria* 
tests  of  mental  skills  may  be 
specified  and  developed? 

2.3.2. 3  Does  the  system  accurately  iden¬ 
tify  the  physical  skills  t'iat 
BIFV  gunners  must  acquire  to 
perform  their  tasks,  so  that 
appropriate  tests  of  physical 
skills  may  be  specified  and 
developed? 

2. 3. 2. 4  Does  the  system  accurately  iden¬ 
tify  the  attitudes  that  BIFV 
gunners  roust  acquire  to  perform 
their  tasks,  so  that  appropriate 
tests  of  attitude  may  be  speci¬ 
fied  and  developed? 

2.3.3  Does  the  system  accurately  iden¬ 
tify  the  conditions  for  learning 
associated  with  the  BIFV  gunner 
tasks,  so  that  appropriate  tests 
may  be  specified  and  developed 
for  assessing  candidate  gunners' 
qualifications? 

2. 3. 3.1  Does  the  system  accurately  iden¬ 
tify  BIFV  gunners'  prerequisite 
knowledge,  so  that  appropriate 
tests  of  the  knowledge  prerequi¬ 
sites  may  be  specified  and 
developed? 


0  The  task  analytic  methods  applied  to 
the  BIFV  gunner  tasks 

0  The  scope  of  the  task  analyses  con¬ 
ducted  (i.e.,  factors  determined  in 
the  analyses) 

0  The  completeness  of  the  BIFV  gunner 
task  analyses 

6  The  accuracy  of  the  BIFV  gunner  task 
analyses 

0  Methods  employed  to  identify  the 
domains  of  learning  involved  in  the 
BIFV  gunner  tasks 

0  The  domains  of  learning  identified 
for  each  BIFV  gunner  task 


0  Methods  employed  to  identify  the 
the  knowledge  requirements  associ¬ 
ated  with  BIFV  gunner  tasks 
0  The  BIFV  gunner  task  knowledge 
requirements  identified 


6  Methods  employed  to  identify  the 
mental  skills  requirements  associ¬ 
ated  with  BIFV  gunner  tasks 
0  The  BIFV  gunner  task  mental  skills 
requirements  identified 


0  Methods  employed  to  identify  the 
physical  skiili1  requirements  associ¬ 
ated  with  BIFV  gunner  tasks 
0  The  BIFV  gur.  er  task  physical  skills 
requirements  ic.«»ntified 


0  Methods  employed  to  identify  the 
attitudinal  requirements  associated 
with  BIFV  gunner  tasks 
0  The  BIFV  gunner  task  attitudinal 
requirements  identified 


0  Methods  employed  to  identify  the 
conditions  for  learning  the  BIFV 
gunner  tasks 

0  Types  of  conditions  for  learning 
identified  for  each  BIFV  gunner  task 


°  Methods  employed  to  identify  the  pre¬ 
requisite  knowledge  required  for 
learning  the  BIFV  gunner  tasks 
0  Knowledge  items  identified  as  pre¬ 
requisite  for  learning  the  BIFV 
gunner  tasks 


Figure  22  (Continued) 
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PART  3:  Issues  and  Attributes  of  "Products"  (Subsets  9-14) 


Subset  No.  10:  "Stated  Performance  Objectives" 


ttributes 


3.2  Does  the  system  produce  accurate 
and  complete  statements  of  the 
performance  objectives  that  BIFV 
gunners  must  achieve  in  order  to 
be  qualified  for  that  job,  so 
that  tests  of  gunners'  qualifi¬ 
cations  may  be  specified? 


#  The  BIFV  gunner  performance  objec¬ 
tives  that  are  stated 
0  The  format  of  the  statements  of 
BIFV  gunner  performance  objectives 
°  The  completeness  of  the  performance 
objective  statements 
°  The  accuracy  of  the  performance 
objective  statements 


3.2.1  Does  the  system  produce  exact 
definitions  of  what  the  gunner- 
learner  are  expected  to  achieve 
through  training,  so  that  tests 
of  thoe  achievements  may  be 
developed? 

3.2. 1.1  Does  the  system  produce  correct 
specifications  of  the  capabili¬ 
ties  that  gunner-learners  are 
expected  to  achieve? 


•  The  gunner-learner  achievements 
that  are  defined 

°  The  factors  included  in  the  defini¬ 
tions  of  gunner-learner  achievements 


0  Capabilities  specified  for  achieve¬ 
ment  by  BIFV  gunner-learners 


3.2. 1.2  Does  the  system  produce  correct 
specifications  of  the  actions 
that  the  gunner-learners  are  to 
execute  to  demonstrate  the 
capabilities? 


°  Actions  specified  for  execution  by 
BIFV  gunner-learners 
0  Observability  of  the  actions 


3. 2. 1.3  Does  the  system  produce  correct 
specifications  of  the  objects 
that  are  to  result  from  the 
execution  of  the  actions  by  the 
gunner-learners? 

3. 2. 1.4  Does  the  system  produce  correct 
specifications  of  the  circum¬ 
stances  under  which  the  gunner- 
learners  are  to  execute  the 
actions  to  produce  the  objects? 

3. 2. 1.5  Does  the  system  produce  correct 
specifications  of  the  tools  and 
equipment  BIFV  gunner-learners 
are  to  use  in  executing  the 
actions? 

3. 2. 1.6  Does  the  system  produce  correct 
specifications  of  the  con¬ 
straints  to  be  imposed  upon  the 
execution  of  the  actions  by  the 
gunner-learners? 

3. 2. 1.7  Does  the  system  produce  correct 
specifications  of  the  criteria 

to  be  used  to  judge  the  adequacy 
of  the  objects  resulting  from 
the  actions  executed  by  the 
gunner-learners? 


0  Objects  specified  as  results  of  the 
action  executions  by  BIFV  gunner- 
learners 

°  Measurability  of  the  objects 


®  Circumstances  specified  for  execu¬ 
tion  of  the  actions  by  BIFV  gunner- 
learners 


°  Tools  and  equipment  specified  for 
use  in  the  execution  of  the  actions 
by  BIFV  gunner-learners 


°  Constraints  specified  for  the 
execution  of  the  actions  by  the 
BIFV  gunner-learners 


0  Criteria  specified  as  standards  of 
adequacy  for  the  objects  resulting 
from  the  gunner-learners'  actions 


Figure  22.  (Continued) 
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PARI'  3  (Continued) 


Subset  No.  13:  "Tests" 

Issues  Attributes* 


3.5  Does  the  system  produce  appro¬ 
priate  tests  for  administration 
to  BIFV  gunner-learners? 


3.5.1  Does  the  system  produce  appro¬ 
priate  tests  for  assessing  the 
qualifications  of  candidate  BIFV 
gunner-learners? 

3. 5. 1.1  Do  the  candidate  qualification 
tests  provide  adequate  measures 
of  the  candidates'  prerequisite 
abilities? 


3. 5. 1.2  Are  appropriate  procedures 
specified  for  applying  the 
qualification  tests  to  BIFV 
gunner  candidates? 


3. 5. 1.3  Are  appropriate  standards  for 
qualification  as  BIFV  gunner 
specified  for  the  candidate 
qualification  tests? 

3.5.2  Does  the  system  produce  appro¬ 
priate  tests  for  assessing 
requirements  for  tailoring  the 
instruction  to  accepted  BIFV 
gunner-learners? 

3. 5. 2.1  Do  the  tailoring  tests  provide 
adequate  pre-traJning  measures 
of  accepted  BIFV  gunner- 
learners'  abilities? 


3. 5. 2. 2  Are  appropriate  procedures  spe¬ 
cified  for  applying  the  tailor¬ 
ing  tests  to  accepted  BIFV 
gunner-learners? 


3. 5. 2. 3  Are  appropriate  standards  for 
tailoring  instruction  specified 
for  the  BIFV  gunnery  tailoring 
tests? 

3.5.3  Does  the  system  produce  appro¬ 
priate  tests  for  measuring  what 
gunner-learners  have  achieved 
as  a  result  of  BIFV  gunnery 
training? 


0  BIFV  gunner-learner  tests  prepared 

0  BIFV  gunnery  testing  applications/ 
requirements  for  which  tests  are 
prepared 

0  BIFV  gunner  behaviors  for  which  the 
tests  are  prepared 

0  Relevance  of  the  tests 

°  Candidate  gunner-learner  qualifica¬ 
tions  for  which  tests  are  prepared 

0  Tests  prepared  for  assessment 
of  those  qualifications 

0  Domains  of  learning  addressed 
by  the  tests 

0  Actions  elicited  by  the  tests 

0  Objects  intended  to  result  from  the 
actions  elicited  by  the  tests 

0  Circumstances  specified  for  the 
tests 

0  Tools  and  equipment  specified  for 
the  tests 

0  Constraints  specified  for  the  tests 

0  Other  aspects  of  applications  pro¬ 
cedures  specified  for  the  tests 

0  Specific  standards  specified  for 
the  qualification  tests 


0  BIFV  gunnery  abilities  for  which 
tailoring  tests  are  prepared 

°  Tests  prepared  for  assessment  of 
needs  for  tailoring  instruction  to 
fit  gunner-learners'  abilities 

0  Domains  of  learning  addressed  by 
the  tests 

0  Actions  elicited  by  the  tests 

°  Objects  intended  to  result  from  the 
actions  elicited  by  th.j  tests 

0  Circumstances  specified  for  the  tests 

0  Tools  and  equipment  specified  for 
the  tests 

0  Constraints  specified  for  the  tests 

0  Other  aspects  of  applications  pro¬ 
cedures  specified  for  the  tests 

0  Specific  standards  specified  for 
the  tailoring  tests 


0  BIFV  gunnery  abilities  for  which 
achievement  tests  are  prepared 
°  Tests  prepared  for  assessment  of 
BIFV  gunner-learners'  achievements 


Figure  22.  (Concluded) 
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to  BIFV  gunnery  operations?  Are  there  any  essential  skills  that  haven't  been 
identified  in  this  analysis?  Are  there  any  skills  that  the  designers  have 
labelled  "physical"  that  really  are  "mental?"  Based  on  these  and  similar 
considerations,  the  evaluator  can  form  a  well  reasoned  judgment  about  the 
accuracy  of  the  designers'  approach  to  identifying  physical  skills  requirements. 
That  reasoned  judgment  will  be  based  on  measures  applied  to  the  two  concrete 
attributes,  viz.,  the  analytic  methods  of  skills  identification  and  the  identified 
skills  themselves. 


The  evaluative  issues  thus  lead  to  measurable  attributes,  and 
the  attributes  in  turn  lead  to  measures  of  performance  and  effectiveness.  The 
question  is:  How  does  all  this  "leading"  proceed?  How  does  one  discern  the 
attributes  in  the  issues,  and  how  does  one  derive  the  measures  from  the 
attributes? 


No  precise  or  even  preliminary  guidelines,  algorithms  or 
heuristics  for  extracting  attributes  from  issues  have  thus  far  been  developed. 
A  major  purpose  of  this  trial  application  of  system  performance  taxonomization 
to  evaluating  BIFV  Gunnery  testing  specifications  is  to  generate  an  experiential 
base  from  which  such  guidelines  can  begin  to  be  derived.  But  in  very  general 
terms,  the  analyst's  thought  process  in  searching  for  the  attributes  of  a 
particular  issue  relevant  to  testing  specification  can  be  sketched  this  way: 

"Here  is  an  issue  that  bears  on  how  well  the  BIFV  Gunnery 
Design  Subsystem  can  define  (or  has  defined)  gunner  testing 
specifications.  I'm  supposed  to  suggest  a  way  of  answering  the 
question  posed  in  this  issue.  First,  I  have  to  ask  myself: 
Which  of  the  subsystem's  people,  resources/equipment,  and 
procedures  are  involved  in  this  issue?  Next,  what  qualities  or 
characteristics  do  those  involved  people,  resources/equipment, 
and  procedures  have  to  possess,  or  produce  in  their'  work,  if 
this  issue  is  to  be  resolved  affirmatively?  Finally,  what 
concrete,  observable  factors  associated  with  those  people, 
resources/equipment,  procedures,  or  their  work  can  I  examaine 
to  determine  whether  those  qualities  or  characteristics  are 
present?" 

The  concrete,  observable  factors  uncovered  by  the  analyst  are 
the  attributes  associated  with  the  issue  in  question.  The  mental  process 
outlined  above  was  applied  to  each  of  the  119  performance-related  issues 
deemed  relevant  to  evaluation  of  BIFV  gunner  testing  specifications.  A  sample 
of  the  outcome  is  the  set  of  attributes  presented  along  with  the  issues  in 
Figure  22. 


If  the  reader  will  accept  the  attribute  set  given  in  Figure  22  as 
being  at  least  illustrative  of  the  bases  from  which  the  evaluative  issues  can  be 
addressed,  attention  can  now  turn  to  the  derivation  of  measures  from 
attributes.  A  measure,  in  most  general  terms,  is  a  judgment  or  appraisal 
about  the  thing-being-measured .  Measures  applied  to  system  atti  ibutes 
derived  from  various  taxa  of  performance  thus  are  judgments  or  appraisals  of 
performance.  Each  such  measure  contributes  some  small  bit  of  wisdom  or 
insight  about  the  total  performance  of  the  system-being-measured.  Every 
measure  "looks"  at  its  attribute  in  its  own  unique  way,  and  "weighs"  the 
attribute  on  its  own  unique  scales.  Collectively,  the  measures  are  intended  to 
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determine  whether  the  attribute  is  "good  enough"  to  meet  the  demands  of  the 
performance  requirement  from  which  it  derives.  Individually,  each  measure 
focuses  on  some  particular  aspect  or  dimension  of  "goodness." 

How  does  one  measure  whether  a  set  of  knowledge  is  "good 
enough."  First,  one  specifies  or  states  exactly  what  is  known,  i.e.,  the 
contents  of  the  set  of  knowledge.  Next,  one  identifies  missing  elements  in  the 
set  of  knowledge,  i.e.,  facts  or  other  information  that  should  be  known,  but 
aren't.  Finally,  one  identifies  inaccuracies  in  the  set  of  knowledge,  i.e., 
supposed  "facts"  and  other  information  that  are  not  true.  This  is  a 
roundabout  way  of  saying  simply  that  a  person's  knowledge  of  a  given  subject 
might  be  "bad"  in  any  of  three  ways: 

0  the  person  might  know  nothing  about  the  subject;  or 

0  might  know  some  things  but  not  others;  or 
might  "know"  some  things  wrong. 

A  combination  of  the  last  two  is  also  possible. 

In  the  specific  context  of  the  Design  Subsystem's  knowledge  of 
the  BIFV  Gunner  Tasks,  the  evaluator  would  need  to  determine,  first,  whether 
the  subsystem  knows  anything  about  any  of  the  tasks;  second,  whether  some 
tasks  are  unknown  to  the  subsystem;  and,  third,  whether  some  of  the  "known" 
tasks  are  in  fact  not  tasks  required  of  BIFV  gunners.  Each  of  those 
determinations  provides  a  separate  judgment  or  appraisal  of  the  goodness  of 
the  scope  of  information  the  Design  Subsystem  proposes  to  use  as  the  basis  for 
establishing  BIFV  gunnery  performance  objectives.  That  is,  each  is  a  separate 
measure  of  that  one  attribute  of  the  system's  potential  for  establishing 
objectives.  The  BIFV  Task  Force,  in  fact,  has  assembled  a  very 
comprehensive  task  data  base  which  helps  to  meet  the  Potentiality  aspects  of 
this  exercise. 


Consider  another  example  in  the  context  of  this  same 
performance  potentiality.  Along  with  having  good  information  about  gunnery 
tasks,  the  Design  Subsystem  must  have  people  who  are  qualified  to  do  the  job 
of  establishing  performance  objectives.  The  staff's  experience  and 
qualifications  thus  constitute  another  attribute  of  the  system's  ability  to 
produce  well  founded  objectives.  If  their  qualifications  aren't  "good  enough," 
thr  '^stem's  ability  to  establish  objectives  also  won't  be  good  enough. 

How  does  one  measure  the  "goodness"  of  an  individual's  or 
team's  qualifications  for  doing  a  particular  job?  When  the  job  is  relatively 
simple  and  of  short  duration,  a  candidate's  qualifications  sometimes  can  be 
measured  by  requesting  him  or  her  to  actually  do  the  work  in  question,  on  a 
sample  basis.  A  typing  test,  for  example,  often  is  administered  to  candidates 
for  a  secretarial  position.  Then,  actual  job  performance  measures  (e.g., 
words  typed  per  minute,  percentage  of  errors,  etc.)  can  be  applied  to  the 
sample  work  and  used  as  job  qualification  measures.  However,  when  the  work 
is  complex,  cerebral  and  non-routine,  this  approach  may  not  be  practical.  In 
such  cases,  clinical  judgments  usually  are  employed  to  appraise  the  candidate's 
qualifications.  Subject-matter  experts  review  resumes  of  the  available 
candidates'  experience  and  training  relevant  to  the  job  to  be  filled,  interview 
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personal  references  named  by  the  candidates,  perhaps  examine  the  products 
delivered  by  the  candidates  during  previous  assignments  similar  to  the  job  to 
be  filled,  and  maybe  interview  the  candidates  themselves  to  obtain  greater 
insight  concerning  the  skills  and  knowledge  they  could  bring  to  the  job.  Each 
expert  reviewer  then  produces  an  independent  rating  of  each  candidate,  based 
on  the  reviewer's  perception  of  the  candidate's  strengths  and  weaknesses. 
Such  ratings  often  are  placed  on  an  interval  or  ordinal  scale.  Then,  the 
expert  reviewers  meet  as  a  group,  report  and  explain  their  individual  ratings, 
deliberate  on  and  debate  the  issues  raised  concerning  each  candidate,  and  form 
a  consensus  rating  of  the  qualifications  of  each. 

This  clinical  approach  can  be  applied  to  measure  the  Training 
Design  staff's  qualifications  for  establishing  gunnery  performance  objectives. 
The  appropriate  measures  would  include  the  consensus  rating  of  each 
individual  proposed  for  assignment  to  the  task  of  establishing  objectives,  and 
an  overall  consensus  rating  of  the  total  proposed  staff  as  a  team  for  handling 
that  task.  Both  types  of  consensus  ratings  should  include  or  be  augmented 
with  detailed  explanations  of  all  identified  personnel  deficiencies  bearing  on  the 
ability  to  establish  objectives. 

It  is  relatively  easy  to  suggest  measures  for  any  given 
attribute.  Unfortunately,  it  is  quite  something  else  to  devise  a  general 
procedure  for  doing  so  in  all  cases.  Within  the  scope  of  this  current  study, 
measures  have  been  suggested  for  all  of  the  issues  and  attributes  of  the  four 
taxonomic  subsets  of  Figure  22.  These  measures  are  arrayed  in  Figures  23 
through  26.  Each  suggested  measure  has  something  of  value  to  say  about  the 
"goodness"  or  "badness"  of  the  attribute  to  which  it  relates.  The  measures 
suggested  for  each  given  attribute  collectively  try  to  say  everything  that  is 
pertinent  to  the  "goodness"  or  "badness"  of  that  attribute.  Nevertheless,  it  is 
quite  possible  (and  probably  very  likely)  that  the  measures  sets  shown  in 
Figures  23  through  26  will  undergo  significant  revision  after  a  critical  review 
by  subject-matter  experts. 

3 .  Concluding  Remarks  Concerning  the  Two  Trial  Applications  of  the 
APM  to  Measurement  of  System  Performance/Effectiveness 


Clearly,  much  remains  unproven  at  the  conclusion  of  these  trial 
applications.  The  final  products  are  merely  partial  sets  of  measures  for 
assessing  very  narrow  aspects  of  the  work  required  of  the  BIFV  and  its 
Gunnery  Training  Design  Subsystem.  Most  glaringly,  those  measures  are 
untried.  No  one  has  actually  attempted  to  apply  them,  or  even  to  verify  that 
the  information  needed  to  generate  them  can  be  obtained.  Those  measures, 
moreover,  derive  from  attributes  that  also  are  unproven.  No  attempt  has  even 
been  made  to  demonstrate  convincingly  that  the  attributes  stated  really 
constitute  all  of  the  factors  pertinent  to  each  evaluative  issue,  and  only  those 
factors. 


What,  then,  has  been  demonstrated?  If  nothing  else,  sets  of 
measures  have  been  produced  that  definitely  are  issues-oriented.  It  has  been 
possible  to  proceed  from  a  framework  of  basic  performance  requirements  to  a 
statement  of  exactly  what  requirements  apply  to  a  given  measurement 
application.  It  has  been  shown  that  such  requirements  can  be  expressed  in 
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Hierarchy  Number  1;  BIFV  gunnery  design  subsystem  potential  for 
establishing  performance  objective* 


Issues 

1.2  Ability  to  Insure  that  ail  BIFV  gunnery 

performance  objectives  will  be  identi¬ 
fied  so  that  they  may  be  addressed  in 
_ gunner  testing  specifications _ 


Attributes 


Measures 


Scope  of  information 
available  concerning 
BIFV  gunners'  tasks 


Staff  experience  and 
qualifications  in 
establishing  perfor¬ 
mance  objectives 


Planned  approach  to 
establishing  BIFV 
gunnery  performance 
objectives 


Resources  allocated 
to  establishing  BIFV 
gunnery  performance 
objectives 


A.  BIFV  gunners'  required  tasks  that  are  known  (listing  and  description 
of  each  such  task) 

B.  BIFV  gunners'  required  tasks  that  are  unknown  to  the  Design  Sub¬ 
system  (listing  of  each) 

C.  Tasks  that  actually  are  not  required  of  BIFV  gunners,  but  which  mis¬ 
takenly  are  classified  as  required  by  the  Design  Subsystem  (listing 
and  description) 

D.  Reviewers'  cumulative  ratings  of  each  Design  staff  member's  training, 
experience  and  previous  performance  in  establishing  performance 
objectives  for  training  (using  a  specified  point  allocation  system 

for  training,  experience,  etc.) 

E.  Reviewers'  overall  assessment  of  total  Design  staff's  qualifications 
(based  on  specified  method  of  accumulating  individual  members' 
ratings) 

F.  Essential  steps  missing  from  the  plans  (listing  and  description  of 
each  such  step) 

C.  Inessential  steps  included  in  the  plans  (listing  and  description  of 
each) 

H.  Inconsistencies/deficiencies  in  the  sequencing  of  steps  in  the  plans 

I.  Reviewers'  cumulative  ratings  of  the  soundness /workability  of  the 
plans  (using  a  specified  point  allocation  system) 

J.  Percentage  of  required  person-hours  proposed  for  allocation  to 
establishing  objectives 

K.  Percentage  of  required  materials/goods  proposed  for  allocation  to 
establishing  objectives 

L.  Percentage  of  required  support  services  proposed  for  allocation  to 
establishing  objectives 

M.  Ratio  of  dollar  value  of  total  resources  proposed  for  allocation  to 
establishing  objectives  to  dollar  value  of  total  resources 
actually  required 


1.2.1  Ability  to  insure  that  all  things  BIFV" 
gunners  will  be  able  to  do  after  train¬ 
ing  will  be  stated  so  that  tests  of  all 
required  abilities  may  be  specified 


Elements  or  factors 
of  gunner  abilities 
intended  to  be 
stated 


A.  Essentia]  elements  of  abilities  intended  to  be  included  in  the  state¬ 
ments  of  BIFV  gunner  abilities  (listing  and  description  of  each  such 
type  of  element) 

B.  Inessential  elements  intended  to  be  included  in  the  statements  of 
BIFV  gunner  abilities  (listing  and  description  of  each  type) 

C.  Essentia]  elements  of  abilities  not  intended  to  be  included  in  the 
statements  of  BIFV  gunner  abilities  (listing  and  description  of 
each  type 


Figure  23.  Sample  Measures  Hierarchy  Number  1 
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Hierarchy  Number  2i  BIFV  gunnery  design  subsystem  process  of 
'  analyzing  tasks  selected  for  training 


2.3 


Issues 

Conduct  of  anaiyais  of1  BiFv  gunner 
tasks,  so  that  appropriate  tests  of 
those  tasks  may  be  specified 


Attributes 

Measures 

Task  analytic 

A. 

Deviations  between  the  methods  actually  applied  and  the  methods  spe¬ 

methods  applied  to 

cified  in  the  plans  for  analyses  (listing  and  description  of  each 

the  BIFV  gunner 

such  deviation) 

tasks 

B. 

Aspects  or  elements  of  the  r,:ethods  applied  that  are  irrelevant  to  or 
inconsistent  with  the  analytic  purposes  (listing  and  description 
of  each) 

C. 

Aspects  or  elements  missing  from  the  methods  applied  that  are  essen¬ 
tial  for  the  analytic  purposes  (listing  and  description  of  each) 

D. 

Types  of  data  needed  for  the  methods  that  were  not  obtained  (listing 
and  description  of  each  data  deficiency) 

The  scope  of  the 

E, 

Factors  supposed  to  be  determined  from  the  task  analyses  that 

BIFV  gunner  task 

actually  were  not  determined 

analyses  conducted 

F. 

Factors  irrelevant  to  the  analytic  purposes  that  were  determined 
in  the  task  analyses 

Completeness  of  the 

G. 

Relevant  BIFV  gunner  tasks  to  which  the  analytic  methods  were  not 

BIFV  gunner  task 

applied 

analyses  conducted 

H. 

Factors  missing  from  the  analyses  applied  to  other  relevant  BIFV 
gunner  tasks 

Accuracy  of  the 

I. 

Inaccuracies  in  the  input  data  obtained  for  application  of  the 

BIFV  gunner  task 

analsytic  methods 

analyses  conducted 

J. 

Specific  misinterpretations  of  the  analysed  data 

K. 

Specific  aspects  or  elements  of  the  analytic  methods  that  were 
misapplied 

2.3.2  Classification  of  BIFV  gunner  tasks 

in  terms  of  the  domains  of  learning 
they  involve 


Methods  employed  to 
identify  the  do¬ 
mains  of  learning 
of  the  BIFV  gunner 

tasks 


A.  Aspects  or  elements  of  the  methods  employed  that  are  irrelevant  to 
or  inconsistent  with  identification  of  the  domains  of  learning 
Involved  in  a  task  (listing  and  description) 

B.  Aspects  or  elements  missing  from  the  methods  employed  that  are 
essentia]  for  identifying  the  domains  of  learning  (listing  and 
deacription) 

C.  Input  data  required  for  the  methods  that  are  missing  or  insufficient 
(listing  and  description) 


The  domains  of 
learning  identified 
for  each  BIFV  gun¬ 
ner  task 


D.  Domains  identified  as  involved  in  a  task  that  actually  are  not  rele¬ 
vant  to  that  task  (listing  and  description  of  each  such  incident) 

E.  Domains  not  identified  as  involved  in  a  task  that  actually  are  in¬ 
volved  in  that  task  (listing  and  description  of  each  such  incident) 


Figure  24.  Sample  Measures  Hierarchy  Number  2 
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Hierarchy  Number  3i  BIFV  gunrtery  design  subsystem  product 
consisting  of  stated  performance  objectives 


3.2 


I««ue« _ 

Production  of  accurate  and  complete 
statements  of  BIFV  gunner  performance 
objectives,  so  that  tests  of  gunners' 
qualifier  dons  may  be  specified _ 


Attributes 


Measures 


The  BIFV  gunner 
performance 

A. 

objectives  that 
are  stated 

B. 

The  format  of  the 
statements  of 

C. 

BIFV  gunner 

performance 

objectives 

D. 

The  completeness 
of  the  statements 

E. 

of  BIFV  gunner 

performance 

objectives 

The  accuracy  of 
the  statements  of 

F. 

BIFV  gunner 

performance 

objectives 

G. 

3.2.1 


The  BIFV  gunner- 
learner  achieve¬ 

A. 

ments  that  are 
defined 

B. 

C. 

D. 

The  factors  in¬ 
cluded  in  the 
definitions  of 

E. 

BIFV  gunner- 
learner  achieve¬ 

F. 

ments 

Statements  that  define  performance  objectives  that  actually  are  not 
needed  for  BIFV  gunnery  (listing  and  description  of  each  such 
statement) 

Performance  objectives  that  actually  are  needed  for  BIFV  gunnery 
that  are  not  addressed  in  any  of  the  statements  produced  (listing 
and  description  of  each  such  objective) 

Essential  elements  of  performance  objective  statements  not  included 
in  the  format  employed  for  stating  BIFV  performance  objectives 
(listing  and  description  of  each  missing  element) 
Inessential/inappropriate  elements  in  the  format  employed  for  stat¬ 
ing  BIFV  performance  objectives  (listing  and  description  of  each 
such  element) 

Elements  missing  from  the  statements  of  particular  BIFV  perfor¬ 
mance  objectives  (tabulation  of  missing  elements  as  a  function  of 
the  stated  objectives) 


BIFV  performance  objective  statements  that  contain  erroneous  or 
inaccurate  elements  (tabulation  of  inaccurate  elements  as  a 
function  of  the  stated  objectives) 

Description  of  the  nature  of  the  inaccuracies,  for  each  stated 
objective 


Production  of  exact  definitions  of 
the  BIFV  gunner-learners'  expected 
achievements  as  a  result  of 
_ the  training 


Achievements  defined  that  actually  are  not  necessarily  expected  of 
BIFV  gunner-learners  during  training  (listing  and  description  of 
each  such  achievement) 

Achievements  not  defined  that  actually  are  expected  of  BIFV 
gunner-learners  during  training  (listing  and  description  of  each 
•uch  achievement) 

Inexact  or  ambiguous  elements  in  the  definitions  of  BIFV  gunner- 
learner  achievements  (listing  and  description) 

Inaccurate  elements  in  the  definitions  of  BIFV  gunner-learner 
achievements  (listing  and  description) 

Factors  Included  In  the  definitions  that  actually  are  irrelevant 
to  the  description  of  intended  achievements  of  BIFV  gunner- 
learners  (listing  and  description) 

Factors  missing  from  the  definitions  that  actually  are  essential 
for  describing  intended  achievements  of  BIFV  gunner-learners 
(listing  and  description) 


Figure  25.  Sample  Measures  Hierarchy  Number  3 
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Hierarchy  Number  4;  BIFV  gunnery  design  subsystem  product 
consisting  of1  teste 


Issues 

3.5  Production  of  appropriate  testa  for 

adminiatration  to  BIFV  gunner-learners 


Attributes 


Measures 


The  BIFV  gunner- 
learner  tests  that 

A. 

are  prepared 

B. 

BIFV  gunnery 
testing  applica¬ 
tion/requirements 
for  which  tests 
are  prepared 

C. 

BIFV  gunner 
behaviors  for 

D. 

which  tests  are 
prepared 

Relevance  of  the 
tests 

E. 

3.5.: 


Candidate  gunner- 
learner  qualifi¬ 

A. 

cations  for  which 
tests  are  prepared 

B. 

C. 

The  tests  pre¬ 
pared  for  assess¬ 

D. 

ment  of  the 
qualifications 

E. 

F. 

G. 

H. 

Prepared  tests  that  address  each  BIFV'  gunnery  performance  objec 
tive  (listing  and  description  as  a  function  of  each  performance 
objective) 

Prepared  tests  that  relate  to  no  BIFV  gunnery  performance  objec¬ 
tive  (listing  and  description  of  each  such  test) 

Reviewers'  ratings  of  the  utility  of  each  prepared  test  for  each 
testing  application /requirement  (yes/no  rating  of  each  test  for 
each  application) 


Listing  of  tests  prepared  for  each  behavior  associated  with  each 
performance  objective 


Deviations  between  the  behaviors  elicited  by  the  tests  and  the 
behaviors  that  actually  are  to  be  addressed  (listing  and  descrip¬ 
tion  of  each  such  deviation) 


Production  of  appropriate  (eats  for 
assessing  the  qualifications  of 
candidate  BIFV  gunner-learners 


Relevant  qualifications  for  which  tests  are  prepared  (listing  and 
description  of  each  such  qualification) 

Relevant  qualifications  for  which  no  tests  are  prepared  (listing 
and  description  of  each  such  qualification) 

Irrelevant  "qualifications*  for  which  tests  are  prepared  (listing 
and  description  of  each) 

Elements  of  each  test  that  deviate  from  the  qualifications  actu¬ 
ally  required  of  acceptable  gunner-lsamers  (listing  and  description 
of  each  such  deviating  element) 

Probability  that  each  test  will  be  passed  by  candidstes  who 

actually  do  not  possess  the  qualification  being  tested 

Probabi  ity  that  each  test  will  be  failed  by  candidates  who 

actually  do  possess  the  qualification  being  tested 

Cumulative  probability  that  an  actually  unqualified  candidate  will 

pass  sufficient  qualification  tests  to  be  accepted  as  a  gunner- 

learner 

Cumulative  probability  that  an  actually  qualified  candidate  will 
fail  sufficient  qualification  tests  to  be  rejected  as  s  gunner- 
learner 


Figure  26.  Sample  Measures  Hierarchy  Number  4 
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terms  that  permit  concrete,  tractable  attributes  to  be  defined.  None  of  the 
attributes  stems  from  any  particular  approach  to  system  design.  All  of  the 
attributes  derive  from  the  performance  requirements  that  are  common  to  all 
such  designs.  This  is  a  critical  point:  it  demonstrates  that  the  APM  at  least 
is  rooted  firmly  in  valid  measurement  issues.  Too  often,  evaluations  of  new 
systems  are  clouded  by  irrelevant  and  unfair  comparisons  with  older  design 
concepts.  The  basic  question  of  whether  the  new  system  does  what  "Tt  is 
supposed  to  do  sometimes  is  overlooked  because  the  evaluators  actually  measure 
how  the  new  system  "looks"  in  comparison  to  older  models.  Conclusions  based 
on  such  evaluations  typically  may  suggest:  "The  system  is  no  good  because 
we've  never  done  it  this  way  before."  That  design-oriented  philosophy  of 
measurement  suppresses  technological  breakthroughs.  In  the  training  sphere, 
for  example,  it  prevents  modern  and  effective  methods  of  work  site, 
learner-tailored  instruction  from  playing  their  appropriate  roles  in  the  total 
learning  environment,  and  it  masks  the  relative  ineffectiveness  of  the 
traditional,  institutional  methods  of  training. 

It  is  a  step  forward  to  have  shown  that  a  measurement  scheme  can 
be  kept  free  of  artificial  judgments  of  design.  Use  of  the  APM  has  forced 
rather  complete  taxonomies  of  performance  requirements  to  be  considered  in 
measurement  applications,  more  complete  than  a  typical,  less  structured 
analysis  would  provide.  It  has  also  demonstrated  the  existence  of  generalized 
and  also  useful  tax,  an  essential  ingredient  in  constructing  a  "systems 
application  science."  Though  much  remains  to  be  done  to  insure  that  the  best 
measures  always  are  identified,  it  has  been  shown  that  measures  can  be 
identified  for  these  performance-oriented  attributes. 
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III.  APM  APPLICATION  PROCEDURES 


A.  General 


The  two  intended  applications  of  the  APM  are  to  specify  the  design  of 
human-machine  systems  and  to  define  measures  of  performance  and  effec¬ 
tiveness  for  those  systems.  As  described  previously,  system  design  and 
system  measurement  deal  with  exactly  the  same  issues.  Those  issues  are  the 
constituent  requirements  of  system  performance.  The  designer  and  the 
evaluator  are  (or  should  be)  interested  strictly  in:  the  capabilities  (potential 
performance)  their  system  is  supposed  to  have;  the  activities  (process  per¬ 
formance)  it  is  supposed  to  carry  out;  and  the  goods  and  services  (product 
performance)  it  is  supposed  to  deliver.  The  designer's  job  is  to  see  to  it  that 
the  system  does  in  fact  meet  all  of  its  requirements  for  potential,  process,  and 
product  performance  within  the  environmental  and  design  constraints  imposed. 
The  evaluator's  job  is  to  check  on  how  well  the  designer's  job  is  done.  Their 
jobs  must  be  performed  as  an  interactive  process. 

To  apply  the  APM,  the  designer  and  evaluator  should  follow  the  steps 
depicted  in  Figure  27.  In  Step  1,  the  analyst  is  required  to  define  the 
purpose  of  this  analysis,  in  terms  that  are  as  specific  and  confining  as 
possible.  This  information  will  be  used  later  in  helping  to  assess  which  of  the 
many  system  performance  items  (which  may  be  presented  to  the  analyst  from  a 
computer  data  base)  are  relevant  to  the  present  analysis.  Specificity  and 
narrowness  of  limits  in  defining  the  analytic  purpose  can  help  keep  the 
ultimate  number  of  design  specifications  or  evaluation  measures  to  a  minimum. 

Step  2  of  the  procedure  requires  the  analyst  to  identify  the  overall  class 
of  systems  to  which  the  system  of  interest  belongs.  The  purpose  of  this 
designation  is  to  permit  the  computer  to  aid  the  user  by  presenting  for  review 
any  known  performance  items,  attributes,  design  specifications  and  measures 
that  are  stored  in  its  data  base  from  previous  analytic  efforts. 

Step  3  allows  the  user  to  identify  the  specific  system  of  interest  so  that, 
for  computer-aided  analyses,  the  proper  identification  automatically  appears  on 
all  displays  and  printouts.  This  serves  as  both  a  record  and  a  reminder  of 
the  application  at  hand. 

In  Step  4,  the  analyst  is  required  to  consider  all  the  other  "systems" 
which  interact  with  the  one  of  central  interest.  Interactions  with 
suprasystems,  collateral  systems  and  subsystems  call  the  analyst's  attention  to 
the  fact  that  the  system  of  interest  must  provide  outputs  to,  and  use  inputs 
from,  external  entities.  The  requirements  of  those  external  entities  thereby 
have  an  influence  on  performance  requirements  being  developed  for  the  system 
of  interest.  This  step  is  not  presently  implemented  on  the  computer-aided 
APM  demonstration  package. 

Step  5  requires  the  user  to  construct  system  performance  taxonomies  in 
accordance  with  the  15-cell  (5  Aspects  x  3  Levels)  matrix  referred  to 
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Figure  27.  General  APM  Application  Procedures 
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previously  as  the  Systems  Taxonomy  Model  (STM)  portion  of  the  APM . 
Because  of  the  voluminous,  equivocal  and  vital  nature  of  this  step,  it  is  of 
special  importance  that  the  analyst  be  given  systematic  directions  and  a  large 
data  base  of  existing,  representative  taxa  to  aid  in  its  satisfactory  completion. 
In  recognition  of  that  need,  the  computer-aided  APM  demonstration  package 
and  this  report  (appendices)  contain  comprehensive  performance  taxa  for 
several  of  the  six  defined  training  subsystems  (for  design,  enabling  and 
delivery,  but  not  for  command,  logistics  and  emplacement).  The  number  of 
performance  items  that  are  retained  as  relevant  to  a  given  analysis  will  depend 
upon  the  purpose  of  that  analysis  (see  Step  1). 

The  task  in  Step  6  is  to  identify  those  system  attributes  that  are  both 
reflective  of  each  performance  requirement  and  susceptible  to  specification  or 
measurement  of  some  kind.  Here,  too,  the  user  can  be  aided  by  having  the 
opportunity  to  review  existing,  representative  attributes  that  a  have  been 
derived  from  the  system  performance  measures.  The  computer-aided  APM 
demonstration  package  and  this  report  (appendices)  contain  a  sample  of  such 
representative  attribute  lists  for  training  systems. 

Step  7  is  the  point  in  the  APM  where  the  analyst  decides  what  will  be 
done  with  the  relevant  performance  attributes  that  have  been  identified.  It  is 
here  that  the  model  divides  into  one  procedure  for  developing  der  .gn  speci¬ 
fications  and  another  procedure  for  developing  measurement  specifications. 

In  Step  8,  the  user  acts  on  the  choice  to  identify  design  specifications 
for  the  relevant  performance  attributes.  Ideally,  the  analyst  will  state 
objective,  operational  criteria  to  be  achieved  by  the  designers  in  providing  for 
the  required  system  attributes. 

In  Step  9,  the  user  acts  on  the  choice  to  identify  evaluation  measures  for 
the  relevant  performance  attributes.  The  measurement  specifications  can  be 
developed  with  the  aid  of  the  model  and  the  existing  sample  of  representative 
measures  to  be  found  in  both  the  computer-aided  package  and  this  report 
(appendices) . 

Step  10  calls  for  the  user  to  apply  the  results  of  this  analysis  to  the 
system  of  interest,  either  in  aiding  its  evolving  design  or  measuring  its 
operational  performance.  Comparisons  of  design  specifications  with 
performance  measures  can  also  result  in  decisions  to  modify  the  system,  its 
performance  requirements,  its  design  specifications,  its  effectiveness  measures, 
or  all  of  these.  This  must  result  from  the  interactive  design/evaluation 
processes  referred  to  earlier. 

Clearly,  the  cost-effective  application  of  this  model  to  typical  military 
systems  requires  some  kind  of  routinized  procedure,  such  as  the  one  illus¬ 
trated  by  the  APM  demonstration  package  developed  as  a  part  of  the  present 
effort.  The  application  of  that  computer-aided  APM  is  described  next. 

B .  The  Computer-Aided  APM 

The  ultimate  computer-aided  APM  will  help  the  analyst  to  design  or 
evaluate  the  effectiveness  of  any  (conceptual  or  real)  system.  The  analyst  will 
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take  one  of  several  approaches  to  working  with  the  system  of  interest.  If  the 
system  is  totally  different  from  any  existing  system,  he  will  start  out  by 
defining  new  performance  items  (objectives,  functional  purposes,  and 
characteristics).  Then,  the  analyst  will  specify  the  attributes  and  measures 
(or  the  design  specifications)  for  each  of  the  performance  items.  Once  this  is 
accomplished,  he  might  specify  reasons  for  performing  an  analysis  and 
associate  each  of  these  purposes  with  a  set  of  characteristics.  Printing  the 
data  set  in  a  variety  of  formats  as  well  as  editing  the  data  set  for  corrections 
and/or  revisions  will,  of  course,  be  possible. 

If  the  system  of  interest  is  very  similar  to  (but  different  from)  an 
existing  system,  the  analyst  will  be  able  to  review,  on  a  line-by-line  basis, 
each  of  the  performance  items,  design  specifications,  attributes,  and  measures 
for  the  existing  similar  system  and  decide  whether  to  include  these  taxa  in  the 
new  system  unedited,  edit  the  items  for  inclusion  in  the  new  system,  or 
exclude  them  from  the  new  system.  Then,  the  analyst  will  be  able  to  associate 
each  of  the  performance  items  with  appropriate  measurement  purposes,  and 
print  the  data  set  in  a  variety  of  different  formats. 

Probably  the  most  frequent  use  of  the  APM  will  be  to  analyze  and/or 
review  the  design  specifications  for  an  existing  system  (any  system  which  has 
previously  been  entered  into  the  computer  is  an  existing  system).  In  this 
case,  one  might  start  by  examining  a  specific  measurement  purpose  for  the 
existing  system,  including,  excluding,  editing,  or  otherwise  reviewing 
individual  taxa  associated  with  the  measurement  purpose.  Alternatively,  the 
analyst  might  decide  to  review  all  of  the  performance  items  design 
specifications,  attributes,  or  measures;  or  the  analyst  might  decide  to  review  a 
specific  performance  item,  a  specific  attribute,  and/or  a  specific  measure. 
After  completing  this  editing  procedure,  the  analyst  will  have  the  opportunity 
to  print  out  the  results  of  his  analysis. 

Another  application  for  the  APM  would  be  for  an  analyst  to  decide  to 
analyze  part  of  a  system,  but  not  the  entire  system.  The  analyst  might 
decide,  for  example,  to  perform  only  30  or  40  percent  of  the  recommended 
measurements.  In  this  case  he  would  have  the  option  to  tell  the  computer 
which  measures  he  would  or  would  not  perform.  The  computer  would  then  tell 
him  which  performance  items  were  going  to  escape  untested,  so  that  he  could 
review,  once  again,  his  testing  procedures. 

This  ultimate  system  would  be  implemented  on  a  main-frame  computer  with 
adequate  on-line  disk  storage  for  several  systems.  Analysts  would  have  the 
opportunity  to  draw  upon  master  files  of  taxa  so  that  they  could  extract  what 
they  would  need  to  construct  their  systems.  Furthermore,  they  would  have 
the  ability  to  compare  their  system  with  a  host  of  other  systems  to  discover 
various  per  ormance  items  which  they  may  have  inadvertantly  left  out  of  their 
data  sets.  The  ultimate  system  would  be  user-friendly,  offering  help  as 
needed.  It  would  come  with  an  on-line  tutorial  program,  and  taxa  for  many 
systems.  Furthermore,  the  system  would  be  compatible  with  both  touch- 
sensitive  screens  (to  insure  maximal  user-friendliness)  and  keyboard  entry 
devices  (to  avoid  excluding  users  without  touch-sensitive  screens). 

A  demonstrntioi  model  which  illustrates  many  of  the  important  features  of 
the  ultimate  system  for  measurement  (but  not  for  design)  was  programmed  in 


-74- 


PASCAL  on  an  APPLE  II  PLUS  Computer*  System  with  an  80-column  VIDF.X 
board.  While  companion  volumes  to  the  present  report  (Shapiro  et  al.,  1983; 
and  Shapiro,  1983)  provide  tutorial  and  operating  instructions  for  the 
demonstration  system  as  well  as  listings  of  the  actual  PASCAL  code.  Figure  28 
presents  an  overall  schematic  flow  chart  of  the  demonstration  system. 

When  the  computer  is  first  turned  on,  a  title  page  appears  on  the  display 
screen.  As  soon  as  an  analyst  presses  a  button  on  the  computer's  keyboard, 
he/she  can  choose  to  read  instructions  which  explain  the  operation  of  the 
demonstration  system  in  detail.  The  instructions  are  user-friendly  in  that 
they  are  written  with  minimal  computer  jargon  and  the  computer  refers  to  itself 
as  "I"  and  to  the  analyst  as  "you."  The  analyst  may  branch  through  the 
instructions  reading  only  those  which  are  of  interest. 

Upon  completing  the  instructions,  the  analyst  selects  which  class  of 
systems  (e.g.,  Training,  Communications)  to  work  with  (only  Training  is 
implemented).  Then  he/she  selects  a  particular  system  (e.g.,  Bradley 
Infantry  Fighting  Vehicle  Training  System).  Finally,  he/she  selects  a 
subsystem  (e.g.,  emplacement,  delivery,  or  design).  After  the  system  class, 
system,  and  subsystem  are  selected,  a  menu  of  all  available  analytic  pro¬ 
cedures  is  displayed  for  analyst  selection.  The  analyst  might,  for  example, 
choose  to  Add,  Modify,  or  Delete  Measurement  Purposes.  Thus,  he/she  would 
define  a  particular  measurement  purpose  (or  use  an  existing  measurement 
purpose).  Then,  the  analyst  would  have  the  opportunity  to  associate  relevant 
characteristics  with  the  measurement  purpose,  or  dissociate  irrelevant 
characteristics. 

Alternatively,  the  analyst  might  select  to  add,  modify,  or  delete 
performance  items  (objectives ,  functional  purposes,  and  characteristics)  for  the 
5x3  matrix  of  the  system.  The  analyst  could  also  add,  modify,  or  delete 
attributes  and  measures  for  any  performance  item.  Whenever  the  analyst  is 
performing  a  procedure,  he/ she  has  the  option  of  making  a  paper  copy  of  the 
taxa  which  he  is  analyzing  at  a  given  time.  Furthermore,  the  analyst  has  the 
opportunity  to  print  out  all  of  the  taxa  for  the  subsystem  of  interest,  or  all  of 
the  taxa  for  a  given  measurement  purpose.  Samples  of  this  printout  for  the 
design,  enabling  and  delivery  subsystems  appear  in  Appendices  A,  B  and  C, 
respectively.  The  printouts  present  the  subsystem  (or  the  taxa  which  are 
relevant  to  a  given  measurement  purpose  for  the  subsystem)  in  n  hierarchy; 
the  aspects  of  the  system  (Potentialities,  Processes,  Products,  Environmental 
Constraints,  Constraints);  the  relevant  objectives;  and  the  functional  pur¬ 
poses,  characteristics,  attributes,  and  measures  for  each  objective.  Each 
taxon  bears  a  unique  number  which  follows  the  general  hierarchical  plan. 
Other  analytic  procedures  available  include  packing  disk  files  more  efficiently 
(a  basic  housekeeping  function),  reviewing  instructions,  stopping,  or  speci¬ 
fying  the  system  class,  system,  or  subsystem  to  be  used  for  the  remainder  of 
the  session. 

The  demonstration  model  is  user-friendly.  In  addition  to  easy-to-read 
instructions,  help  messages  are  available  in  the  initial  program  where  one 
specifies  which  system  he/she  wishes  to  use,  and  which  analytic  procedure 
he/she  wishes  to  perform.  However,  they  are  not  available  in  other  programs. 
The  demonstration  system  is  also  very  forgiving  of  typing  errors.  Whenever  it 
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catches  an  illegal  response,  it  tells  the  analyst  what  the  legal  responses  are. 
The  system  Hoes,  however,  have  certain  very  strict  limitations.  Most  of  these 
limitations  are  caused  by  the  relatively  slow  processing  speed  of  the  Apple  II 
Plus  computer,  and  the  relatively  limited  core  and  disk  storage  available  on  the 
Apple  II  Plus,  as  well  as  the  attempt  to  avoid  excessive  use  of  machine- 
dependent  code.  A  comparison  of  the  specific  limit  tions  of  the  demonstration 
system  and  a  list  of  features  to  be  available  in  lae  ultimate  system  are 
presented  in  Figure  29. 
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(continued) 


Figure  29.  Comparison  between  the  Ultimate  Computer  Package 
and  the  Demonstration  Package 
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Descriotion 

Ultimate  Package 
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estimate) 
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CODE  EFFICIENCY 

Use  search  routine  which  is  optimal  for  task 
at  hand 
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Single-line  updates 
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display 

TUTORIAL  PROGRAM 

Online 

In  booklet  form 
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Thus,  potential  measures  are  not  neglected. 
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No 

Figure  29.  (Concluded) 
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IV.  RELATIONSHIP  OF  THE  APM  10  OTHER  MODELS* 


A.  Bases  of  Comparison 

There  "re,  of  course,  many  "models"  available  for  the  development  and 
evaluation  of  human-machine  and  training  systems.  Indeed,  there  are  so  many 
such  models  that  it  is  difficult  to  list  them  let  alone  select  an  appropriate 
model  for  an  application.  These  models  tend  (1)  to  be  based  on  a  variety  of 
mathematical  and  descriptive  bases,  (2)  to  be  specific  system-  and/or 
aggregate  systems-oriented ,  (3)  to  concentrate  on  macro-  or  micro-levels  or 
somewhere  in  between,  and  (4)  to  focus  on  parts  of  the  total  human  resources 
in  system  problems.  It  is,  therefore,  not  easy  to  place  the  APM  within  the 
general  framework  of  human-machine  and  training  systems  models  since  that 
framework  is,  at  present,  not  clearly  defined  and  structured. 

One  way  of  looking  at  the  APM  is  within  some  of  the  major  acquisition  and 
development  steps  for  all  military  systems.  In  this  chapter,  six  such  steps 
are  noted  as  follows: 

°  H  i  lan-machine  system  development  procedures 

°  Human-machine  system  development 

°  Training  system  development 

°  Behavioral  Simula t5on  models 

°  Logistic  support  analyses 

°  Test  and  evaluation  methods 

Thus,  the  emphasis  here  is  on  the  possible  use  of  the  APM,  as  compared  with 
other  models,  in  some  of  the  major  events  in  system  development.’  The 
question,  then,  is  not  so  much  the  internal  structure  of  the  APM  insofar  as 
completeness,  coherence,  and  validity  but,  rather,  how  relatively  well  it  can 
be  used  in  a  comparison  with  other  models  and  methods  in  the  technical 
inventory . 

With  this  approach,  certain  utility  features  of  any  model  become 
paramount.  Questions  arise,  such  as:  (1)  How  well  does  the  model  l.t  the 
system  (or  systems)  under  development?  (2)  What  kinds  of  data  are  needed 
for  the  model  to  work  well?  (3)  How  much  time  and  project  resources  are 
needed  to  use  the  model?  (4)  Does  the  model  produce  requirements  and  data 
useful  in  system  development?  and  (5)  What  kinds  of  analytic  skills  are 
needed  to  apply  the  model  (i.e.  ,  can  it  be  used  by  normal  project  people  or 
does  it  require  very  highly  talented  people  who  may  or  may  not  be  available  to 
the  program)?  These  are  all  very  practical  problems  which  seem  to  determine 
to  a  great  extent  whether  or  not  models  are  used  in  the  system  development 
project. 


*This  chapter  was  prepared  by  Dr.  Frederick  A.  Muckier  of  Canyon  Research 
Group,  Inc.  (now  Essex  Corporation),  Westlake  Village,  California. 
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B.  System  Development  Procedures 


The  major  single  document  that  defines  the  procedures  for  the  design  and 
development  of  human-machine  systems  is  MIL-H-46855B  (Department  of 
Defense,  1979)  "Human  Engineering  Requirements  for  Military  Systems, 
Equipment  and  Facilities."  This  general  specification,  and  its  subsequent 
revisions,  has  been  in  effect  since  1968,  and  is  presumably  mandatory  for  all 
military  system  development  projects.  Basically,  it  provides  a  top-level 
procedure  for  analysis,  design  and  development,  and  test  and  evaluation  of  all 
human-machine  military  systems  and  subsystems.  In  practice,  many  variations 
have  been  followed  to  implement  the  specification.  Indeed,  it  was  designed  to 
allow  for  such  variations  consistent  with  differing  weapon  systems  across  the 
military  services.  The  end  goal  of  the  specification  is  to  attempt  to  assure 
that  the  human  element  of  military  systems  is  used  properly  and  effectively. 

The  APM  is  totally  consistent  with  the  general  objectives  of 
Mil,— H— 4685513 .  The  APM  may,  in  fact,  be  considered  an  implementing  and 
detailed  model  of  that  specification.  The  APM  places  primary  emphasis,  in  its 
present  form,  on  system,  subsystem,  and  mission  analysis  (to  use  the 
terminology  of  MII.-H-46855B)  and  stresses  "...  the  functions  that  must  be 
performed  by  the  system  in  achieving  its  mission  objectives  (which)  shall  be 
identified  and  described."  (MIL-H-46855B :  3. 1.1. a)  Two  major  strengths 

may  be  noted  in  the  APM  process: 

°  An  initial  emphasis  on  system  goals  and  objectives  which  forms  the 
framework  for  all  subsequent  analysis,  design,  and  test. 

0  A  major  requirement  to  evaluate  all  system  and  subsystem  levels  of 
potentialities,  processes,  and  products. 

Within  each  system  level,  specific  answers  must  Ve  given  to  the  questions: 
What?  (Objectives)  Why?  (Functional  Purposes  and  How?  (Characteristics) 
This  is  precisely  the  intent  of  the  analytic  steps  in  MIL-H-46855B ;  the  APM  is 
the  most  highly  structured  tool  for  accomplishing  this  objective  currently 
available.  Further,  in  its  computerized  form,  it  assists  the  analyst  in  rapid 
use  of  the  model.  Finally,  the  computerized  APM  is  "user  friendly,"  and  takes 
into  account  a  widely  varying  skill  level  among  potential  users. 

With  respect  to  model  structure,  the  APM  is  consistent  with  standard 
system  engineering  methods  which  have  been  in  use  for  the  past  .30  years 
(cf.  ,  Machol,  1965;  Byrne,  Mullally,  and  Tohery,  1969).  Within  that 
framework,  the  APM  may  be  classified  as: 

°  A  descriptive  model  with,  however,  specific  quantitative  outputs  in 
the  measurement  subsystem;  and 

°  A  static,  multi-level  model  (cf.  Niehaus,  1979)  which  can  give  rapid 
snapshots  of  the  development  status  of  the  system  insofar  as  human 
resources  are  concerned,  as  well  as  system  performance  implications. 
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On  the  latter  point,  a  choice  is  available  to  system  analysts  between  static 
and  dynamic  models.  Dynamic  models  assume  a  time-varying,  scenario-based 
simulation.  At  the  present  time,  the  APM  is  not  constructed  for  this  use 
although  future  development  could  make  it  so.  Further,  no  explicit  use  has 
been  made  of  a  great  number  of  optimization  techniques  (cf. ,  Rouse,  1980) 
that  might  be  used  for  human-machine  systems  problems. 

While  all  systems  models  are  concerned  with  measurement  and  measures, 
the  APM  appears  to  be  unique  in  its  stress  on  the  methods  for  deriving 
specific  and  practical  measures.  Often,  measurement  is  left  to  the  skill  of  the 
analyst  with  the  method  undefined;  APM  requires  a  direct  and  explicit 
development  of  system  and  subsystem  measurement.  As  will  be  noted,  the  data 
estimates  so  derived  will  be  useful  for  many  purposes. 

C.  Human-Machine  System  Development 

The  major  purpose  of  MIL-H-46855B  was  to  give  a  general,  sequential 
method  by  which  human-machine  systems  could  be  developed  with  careful 
emphasis — in  the  total  systems  context — on  the  use  of  the  human  element.  As 
noted,  the  APM  is  a  specific  method  for  doing  exactly  that.  While  there  are  a 
number  of  particular  techniques  for  human  engineering  development  (cf. , 
Meister,  1976),  the  APM  appears  to  be  the  most  comprehensive,  thorough,  and 
detailed  method  to  date.  It  fulfills  the  data  requirements  for  MIL-H-46855B , 
but  goes  beyond  those  requirements  to  specify  training  system  requirements 
and  provides  basic  data  for  logistic  system  support  analyses. 

Much  practical  experience  tends  to  confirm  the  fact  that  systematic 
attention  to  human  resources  within  the  system  development  process  varies 
enormously.  The  range  is  from  very  detailed  and  competent  contributions  to 
ignoring  the  human  element  altogether.  The  reasons  for  this  are  complex  (cf., 
Carr,  et  al.  ,  1980),  but  one  problem  has  been  the  lack  of  methods  that 
significantly  answer  all  of  the  manpower,  personnel  and  training  questions  that 
arise  from  a  new  system. 

One  method  (Mackie,  1979)  has  organized  the  system  development  process 
into  nine  basic  questions  concerning  the  manpower,  training,  and  personnel 
elements  of  the  system: 

1 .  How  do  various  design  concepts  affect  manpower  and  training 
requirements? 

2.  How  do  manpower  and  other  evaluative  criteria  interact? 

3.  What  numbers  and  types  of  personnel  are  associated  with  this 
system? 

4.  What  is  their  projected  availability? 

5.  Which  design  alternative  best  satisfies  all  criteria? 

6.  What  operator  and  maintenance  tasks  are  implied  by  this  design? 

7.  What  are  the  operational  and  maintenance  training  requirements? 
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8.  What  skill  levels  and  numbers  of  operators  are  required? 

9.  What  are  the  estimated  personnel  life-cycle  costs  to  man  the  system? 

The  procedure  also  suggests  the  kinds  of  data  that  would  be  needed  to  answer 
these  questions. 

One  way  of  evaluating  the  APM — or  any  other  model  framework  for  system 
development  application — is  to  estimate  the  degree  to  which  APM  can  answer 
these  questions.  Following  the  order  of  the  questions  as  just  given: 

1.  Alternative  design  concepts  can  be  evaluated  within  successive  runs 
of  the  APM  with  its  major  strength  in  its  present  form  in  personnel 
and  training  estimates. 

2.  Criteria  interaction  assessment  should  be  particularly  strong  since 
the  APM  model  requires  careful  delineation  of  goals  and  objectives 
with  specific  measurement  dimensions. 

3.  There  is  no  clear  path,  at  present,  in  the  APM  to  estimate  numbers 
and  types  of  personnel. 

4.  At  present,  APM  does  not  take  into  account  projected  manpower 
availability,  although  it  certainly  could  be  expanded  to  do  so. 

5.  Trade-off  studies  based  on  explicit  criteria  from  APM  should  be 
particularly  effective  (cf. ,  Askren,  1975). 

6.  Task  analysis  data  are  an  explicit  output  of  the  APM. 

7.  Training  requirements  analysis  is  an  explicit  output  and  strength  of 
the  APM. 

8.  Identification  of  skill  levels  is  a  specific  output  of  the  APM. 

9.  Life-cycle  costing  is  not,  at  present,  an  identified  process  or  output 
for  APM. 

Valuable  outputs  from  the  APM  which  are  not  included  in  these  questions  (and 
which  should  be)  include  logistic  support  analysis  and  specific  test  and 
evaluation  criteria. 

No  current  model,  method,  or  procedure  satisfies  all  of  the  development 
tasks  and  data  that  are  needed  for  system  design  and  development.  At 
present,  the  APM  is  strongest  in  system,  task,  training  and  measurement 
definition  but  appears  to  be  weak  in  responding  to  manpower  and  cost 
questions  associated  with  new  systems. 

D.  Training  System  Development 

The  past  30  years  have  seen  the  most  remarkable  advances  in  training 
technology  in  the  entire  history  of  human  learning.  Part  of  this  is  due  to 
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applications  of  computer  technology,  part  to  empirical  facts  about  optimal 
human  learning,  and  much  to  systematic  methods  of  analyzing  and  developing 
training  systems.  In  a  broad  sense,  the  last  can  be  termed  the  "Systems 
Approach  to  Training"  or  "Instructional  System  Development"  (O'Neil,  1979; 
Vineberg  and  Joyner,  1980).  Instructional  system  development  (ISD) 
methodologies  are  in  use  throughout  all  military  services,  and  are  widely  used 
in  non-military  education  and  training.  Unfortunately,  several  current 
problems  are  apparent  with  the  ISD  technique: 

0  There  is  a  variety  of  methods  for  generating  training  systems  called 
by  the  loose  generic  label  of  ISD; 

0  There  is  no  guarantee  that  any  of  the  ISD  methods  obtain  maxinun 
system  performance  efficiency  or  training  efficiency;  and 

ISD  applications  frequently  come  too  late  in  the  system  development 
cycle  where  potential  major  training  problems  could  be  identified 
before  design  freeze  and  corrected  through  job  and  task  redesign. 

To  date,  the  major  applications  of  the  APM  have  been  on  training  systems 
development.  This  is,  in  effect,  a  test  of  how  well  APM  can  work  to  generate 
training  requirements  and  training  system  data  bases  (see  Appendices  A,  B, 
and  C).  APM  is  particularly  strong  in: 

°  Performance  goals  and  objectives 

°  Need/discrepancy  analysis 

°  Identification  of  job  requirements 

Selection  of  tasks  for  training 
Analysis  of  tasks 

Construction  of  job  performance  measures 
0  Development  of  training  objectives  and  objective  hierarchies 
°  Validation  of  instruction 

0  Logistic  requirements  for  training  systems 

It  appears  to  be  less  useful  in  such  steps  as: 

0  Methods  for  evaluating  optimum  training  efficiency 
0  Formal  cost-effectiveness  evaluations  of  training  systems 

Perhaps  another  way  of  stating  it  is  that  APM  is  currently  emphasizing  the 
"What?"  (Objectives)  and  "Why?"  (Functional  Purposes)  of  new  training 
systems  rather  than  the  "How?"  (Characteristics).  For  the  latter,  additional 
development  is  needed.  There  is,  however,  nothing  inherent  in  the  APM 
concept  that  would  preclude  its  addressing  the  "how"  of  new  training  systems. 
Indeed,  the  APM  is  so  structured  that  the  "how"  would  be  a  natural  output. 

It  would  appear  that  the  major  strength  of  the  APM  at  present  is  with 
respect  to  front-end  analysis  of  training  systems.  As  noted  above,  training 
system  analysis  has  usually  come  very  late  in  the  system  development  cycle, 
and  there  is  much  concern  that  these  analyses  be  conducted  as  a  part  of  the 
system  development  process  (cf.,  Seidel  and  Wagner,  1979).  APM  seems 
particularly  potentially  useful  in  the  initial  steps  of  mission,  function,  task, 
and  job  analysis. 


-84- 


Another  similar  system  under  development  at  this  time  is  the  Early 
Training  Estimation  System  (ETES)  which  will  attempt  "...  to  deal  with  two 
major  deficiencies  in  existing  technologies:  (1)  the  lack  of  a  systematic  tool 
for  describing,  storing,  and  updating  system  concepts  and  for  transmitting 
this  information  to  all  of  the  various  participants  in  the  development/acquisition 
process,  and  (2)  the  lack  of  a  comprehensive  set  of  training  analysis  tools 
which  are  appropriate  for  the  early  phases  of  design."  (O'Brien,  1982). 
ETES  has  three  major  functional  components:  System  Description  Technology, 
Training  Estimation  Aids  and  Procedures,  and  Evaluative  Technology.  It 
appears  to  be  particularly  strong  in  comparative  evaluation  of  instructional 
methods  and  media  and  cost-effectiveness  evaluation.  These  are  areas  in 
which  APM  has  not  moved  into  heavily.  Note  should  be  made  of  the  fact  that 
ETES  is  under  development  for  computer  application  with  a  microprocessor. 

The  APM  and  the  ETES  appear  to  be  complementary.  ETES  can  be  of 
particular  value  to  APM  in  comparative  evaluations  of  media  and  instructional 
methods.  APM  appears  to  have  a  sounder  approach  to  the  generation  of 
system  and  subsystem  objectives  and  development  of  performance  measurement. 
Both  systems  appear  to  generate  the  kinds  of  support  data  for  early  training 
projections  as  described  by  Jorgensen  (1981). 

E.  Behavioral  Simulation  Models 


For  almost  30  years  there  has  been  a  continuing  attempt  to  develop 
quantitative  simulation  models  for  behavioral,  social,  and  systems  processes 
(cf.,  Siegel  and  Wolf,  1969;  Pew,  Baron,  Feehrer,  and  Miller,  1977;  Rouse, 
1980).  Initial  theoretical  attempts  concentrated  on  human  tracking 
performance,  and  have  now  expanded  to  encompass  decision  making  and 
information  processing  at  both  direct  control  and  monitoring  levels.  The 
question  might  be  asked:  What  relevance  do  these  models  have  to  APM? 

The  primary  emphasis  of  these  models  has  been  a  better  understanding 
and  description  of  human  behavior  often  (but  not  always)  in  a  system  context. 
For  the  systems  context  and  tasks  for  which  they  are  appropriate,  these 
models  have  sometimes  provided  very  specific  quantitative  predictions  of  hum  in 
behavior.  This  has  normally  concerned  specific  operator  and  maintenance 
tasks  with  little  concern  for  manpower  and/or  training  implications.  They  are 
normally  time-varying ,  stochastic  models  with  much  concern  about  task 
performance  and  often  human  performance  optimization. 

As  Siegel,  Wolf,  and  Ozkaptan  (1981)  have  pointed  out,  early  in  the 
system  development  cycle  it  may  be  wise  to  use  qualitative  and  deterministic 
methods  before  switching  to  complex,  computer-based,  behavioral  simulations. 
If  that  is  true,  APM  would  in  general  be  the  method  of  choice  early  in  design. 
APM  could  then  subsequently  provide  performance  parameters,  tasks,  and 
measures  for  time-varying,  scenario-driven,  stochastic  simulation  models.  In 
short,  APM  would  describe  the  objectives,  structure,  and  measures  of  the 
system  for  computer  simulation  of  the  system. 
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F.  Logistic  Support  Analyses 


Perhaps  even  to  a  greater  extent  than  training,  logistic  support  analyses 
have  traditionally  been  executed  well  into  the  system  development  cycle  and 
usually  after  design  freeze.  It  is  now  apparent  that  this  can  no  longer  be  the 
case;  early  estimates  of  logistic  requirements  are  essential  since  they  may 
determine  whether  or  not  a  system  is  operationally  feasible.  The  goal,  as 
Weddle  (1983  )  puts  it  is  "...  to  build  supportability  into  a  system  rather  than 
inefficiently  bolting  it  on  after  the  system  is  deployed." 

There  arc  a  very  large  number  of  logistic  support  analyses  models,  two  of 
which  may  exemplify  their  structure  and  purpose: 

1.  Kyle  and  Markisohn  (1974)  have  described  an  Integrated  Facilities 
Requirements  System  for  pilot  training.  It  contained  five  principal 
modules: 

Logistic  support  requirements  for  calculating  total  manpower, 
aircraft  and  fuel  requirements  for  training; 

0  Base  loading  for  assignment  of  students  and  resources  for  each 
phase  of  training; 

Facilities  requirements  for  estimating  specific  permanent 
facilities  for  the  total  pilot  training  program  at  each  base; 

0  Efficiency/deficiency  estimates  to  evaluate  the  training  input 
load  with  facilities;  and 

0  Total  systems  cost  which  attempts  to  predict  the  total  cost  of  all 
resources  required,  including  investment  costs  and  operations 
and  maintenance  costs. 

With  the  exception  of  cost  data,  the  APM  is  designed  to  supply  these  types  of 
logistics  data  (see  specifically  Appendices  B  and  C). 

2.  While  all  services  have  developed  extensive  logistic  support  analyses 
models,  the  Air  Force  has  particularly  pioneered  the  inclusion  of  the 
human  resources  element  in  logistics  (cf. ,  King  and  Askren,  1980). 
They  have  developed  and  tested  a  coordinated  human  resources 
technology  (CURT)  method  for  logistics  analyses  combining: 
maintenance  manpower  modeling,  job  guide  development,  instructional 
system  development,  system  ownership  costing,  and  human  resources 
in  design  trade-offs.  Most  important  is  an  associated  requirement 
for  a  unified  data  base  to  support  logistic  analyses  in  these  areas. 
APM  stresses,  to  a  greater  degree  than  CURT  and  its  derivative 
Navy  system,  Hardman,  system  and  task  performance  objectives  and 
performance  measurement. 

G .  Test  and  Evaluation 

Very  early  in  the  introduction  of  human  resources  work  in  system 
development  stress  was  placed  on  adequate  test  and  evaluation,  particularly  of 
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the  human  element  in  systems  (cf.  ,  Mcister  and  Rabideau,  1965).  APM 
continues  this  tradition  with  a  strong  and  explicit  module  for  test  and 
evaluation.  Although  there  has  been  a  considerable  literature  and  practice 
developed,  little  standardization  has  been  achieved  for  human  resources  test 
and  evaluation  either  in  developmental  or  operational  testing.  Major  techniques 
have  been  developed,  including  the  HEDGE  system  for  the  Army's  Human 
Engineering  Laboratory  and  the  HFTEMAN  (Human  Factors  Test  and  Evaluation 
Manual)  for  the  Navy. 

A  major  Army  test  and  evaluation  system  has  been  the  Human  Resources 
Test  and  Evaluation  System  (I1RTES)  principally  developed  for  operational 
testing  (cf.,  Kaplan  and  Crooks,  1980).  HRTES  assumes  a  prototype 
operational  system  ready  for  operational  test  and  evaluation.  A  test  plan  is 
generated  by  IIRTES,  performance  measures  obtained,  and  the  performance 
evaluated  by  an  application  of  multi-attribute  utility  theory.  A  principal 
feature  is  the  diagnosis  and  evaluation  of  unsuccessful  performance.  In  short, 
when  did  an  error  occur;  why  did  it  happen;  and  what  should  be  done  about 
it? 


In  contrast,  APM  is  directed  toward  developing  systems  where  specific 
system  parameters  are  evolving  in  system  design,  places  particular  emphasis  on 
system  goals  and  objectives,  but  at  the  same  tine  assigns  particular  measures 
to  system  parameters  and  objectives.  These  data,  if  generated,  would  be 
particularly  pertinent  as  critical  inputs  to  the  HRTES.  APM  would  provide  a 
system  analysis  framework  for  the  HRTES  evaluation.  HRTES,  then,  extends 
APM  into  the  domain  of  operational  testing  but,  at  the  same  time,  APM 
provides  a  measurement  framework  for  the  HRTES  technique. 

H .  Summary 

APM  is  a  static,  deterministic  model  principally  useful  for  early  system 
development  and  for  front-end  analysis  of  human  resources  elements  of  military 
systems.  It  is  specifically  valuable  in  deriving  system,  subsystem,  and  human 
performance  objectives  in  a  manner  consistent  with  standard  system 
engineering  methods  as  well  as  the  goals  of  MIL-II-46855B .  No  other  available 
models  have  this  capability.  Outputs  of  the  APM  may  be  used  within  other 
model  frameworks  for: 

0  Stochastic,  time-varying,  scenario-driven  simulations 

°  Training  system  development 

0  Logistic  support  analyses 

0  Human  resources  test  and  evaluation 

It  is,  therefore,  complementary  to  other  models  in  these  areas.  Perhaps  its 
majc*-  weakness  as  a  model  is  a  lack  of  costing  data  which  might  be  essential 
for  early  evaluation  decisions.  In  its  computerized  version  it  allows  for  quick 
development  and  assessment  of  initial  system  goals,  processes,  and  products 
before  actual  system  hardware  and  software  have  been  selected. 
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V.  RECOMMENDATIONS  FOR  FURTHER  DEVELOPMENT  AND  APPLICATIONS 


As  noted  earlier,  the  ultimate  objective  of  this  APM  development  work  is  to 
provide  users  with  a  uniform,  thorough,  adaptive  and  efficient  procedure  to 
aid  in  the  process  of  deriving  the  most  meaningful  design  specification 
requirements,  design  specifications,  evaluation  measurement  requirements  or 
evaluation  measures  for  any  planned  or  existing  human-machine  system 
(particularly  a  training  system).  The  users  of  the  APM  for  design  and 
measurement  purposes  are  expected  to  be  at:  the  U.S.  Army  Infantry  School 
(USAIS) ;  other  schools  of  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC);  the  Directorate  of  Training  Development  (DTD);  the  Directorate  of 
Evaluation  (DOE);  and  the  Directorate  of  Combat  Development  (DCD).  APM 
users  are  also  foreseen  to  include  personnel  with  the  U.S.  Army  Test  Boards, 
as  well  as  the  U.S.  Army  Operational  Test  and  Evaluation  Agency  (USAOTEA) 
and,  of  course,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI).  Among  the  individuals  at  those  agencies  who  are  seen 
as  the  ultimate  users  of  automated  APM  are  training  developers  and  evaluators, 
TRADOC  system  managers  (TSMs),  system  researchers  and  designers,  analysts 
and  planners.  The  APM  project  has  supplied  the  following  products  to  help 
these  users  perform  their  work: 

°  An  initial  analytic  process  model  for  systems  design  and 
measurement . 

0  A  generalizable  taxonomy  of  measurable  attributes  for  the 

surveillance  function  found  in  many  manned  systems. 

°  A  generalizable  model  of  training  systems  and  their  component 

subsystems. 

°  Generalizable  taxonomies  for  the  Design  (curriculum  development), 
Enabling  (trainer  preparation)  and  Delivery  (instruction)  subsystems 
of  training  systems. 

°  A  sample  application  of  the  model  for  deriving  effectiveness  measures 
foi  Learner  Testing  Specifications  within  the  emerging  BlFVTS, 
including  a  generalizable  taxonomy  and  application  procedures. 

°  A  preliminary  review  of  concepts  for  applying  the  APM  in 

establishing  design  requirements. 

°  An  untested  procedural  outline  and  demonstration  package  of  a 
computer-aided  APM  for  deriving  training  systems  measures  (using 
an  Apple  II  Plus  computer) . 

An  annotated  bibliography  of  the  manned  systems  measurement 
literature  (244  documents  cited  and  abstracted). 

This  list  represents  considerable  progress  in  an  area  that  has  been  so  difficult 
to  standardize  for  so  many  years.  The  model  development  is  just  now 
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beginning  to  yield  the  kinds  of  guidelines  and  sample  applications  that  soon 
w;11  allow  it  to  be  learned  and  applied  by  those  who  are  its  potential  users. 
To  help  bring  the  APM  development  process  to  the  point  of  application  by  the 
ultimate  users,  several  tasks  need  to  be  carried  out.  These  are  listed  in 
Figure  30.  It  is  noted  that  the  items  are  not  necessarily  independent  of  each 
other,  but  some  tasks  can  be  seen  as  being  a  part  of  others.  They  are  listed 
separately,  however,  because  they  are  identifiably  distinct  or  limited  in  their 
purpose.  For  convenience,  the  tasks  are  divided  into  four  groups,  according 
to  their  primary  concerns:  data  base  development,  procedures  development, 
applications  and  application  aids,  and  administrative  items  development. 

A .  Data  Base  Development 

1 .  Develop  Performance  Items,  Attributes,  Measures  and  Specification 
Requirements  for  the  Remaining  Training  Subsystems  (Command, 
Logistics  and  Emplacement) 

Currently,  generalized  taxa  for  any  design,  enabling  and  delivery 
subsystem  have  been  developed,  though  their  measures  and  specification 
requirements  remain  in  need  of  completion.  Taxa  for  the  additional  generalized 
training  subsystems  also  should  be  developed.  Their  applicability  to  B1FVTS 
or  other  issues  could  be  verified  through  some  real  world  comparison  or 
evaluation,  by  subject-matter  experts  (SMEs)  or  otherwise.  The  same 
procedures  should  he  applied  as  used  in  prior  taxa  development. 

2 .  Develop  Data  Bases  for  Other  Than  Training  Systems 

This  task  is  intended  to  make  the  APM  useful  for  analyzing 
human-machine  systems  in  addition  to  those  in  the  training  system  class. 
Depending  upon  the  active  work  areas  of  the  using  agency,  generalized 
taxonomies  of  performance  items,  attributes,  measures  and  specification 
requirements  could  be  developed  for  such  system  classes  as  weapons, 
communication,  navigation,  search  and  surveillance,  target  acquisition  and  fire 
control,  command  and  control,  intelligence,  electronic  warfare  or  any  other 
type. 


This  task  should  identify  the  many  (sometimes  subtle)  roles  of  human 
operators  in  the  total  system  measurement  and  evaluation  process.  It  should 
clarify  the  contribution  and  influence  of  operators  in  the  various  system 
attributes  selected  for  specification  or  measurement. 

3 .  Identify  the  Sub-Taxonomies  for  Generally  Stated  Analytic  Purposes 

This  task  is  intended  to  provide  the  user  with  computer-aided 
recommendations  of  appropriate  performance  items  for  certain  stated  purposes 
of  measurement  or  specification.  It  consists  of  a  mapping  process  in  which 
subsets  of  performance  items  are  identified  for  each  generally  stated  analytic 
purpose.  The  APM  would  then  present  the  user  with  those  recommendations, 
if  his  purpose  were  one  of  those  in  storage,  thus  easing  the  user's  task  of 
generating  a  completely  new  subset  of  purpose-relevant  performance  items. 
The  user  would  then  proceed  by  editing  the  recommended  subset  to  suit  his 
needs.  For  reasons  of  efficiency,  this  task  should  be  carried  out  together 
with  Task  8. 
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Data  Base  Development 


1.  Develop  performance  items,  attributes,  measures  and 

specification  requirements  for  the  remaining  training 
subsystems  (command,  logistics  and  emplacement) 

2.  Develop  data  bases  for  other  than  training  systems 

3.  Identify  the  sub-taxonomies  for  generally  stated  analytic 

purposes 

Procedures  Development 

4.  Further  develop  procedures  for  APM  application  to 

measures  derivation 

5.  Further  develop  procedures  for  APM  application  to  system 
design  specification 

6.  Develop  procedures  for  differentiating  specifications  and 
measures  in  terms  of  weighting  factors,  delay  tolerance 
and  criticality 

7 .  Modify  procedures  and  programs  for  use  with  a  computer 
large  enough  to  provide  the  user  assistance,  data  base  and 
processing  capacity,  and  speed  consistent  with  the 
expanded  capabilities 

8.  Improve  procedures  for  specifying  new  analytic  purposes 

Applications  and  Application  Aids 

9.  Train  users  to  apply  the  APM 

10.  Apply  the  APM  to  the  measurement  of  an  existing  Army 
system 

11.  Apply  the  APM  to  the  specification  of  a  new  Army  system 

12.  Provide  the  user  with  a  computer-generated  assessment  of 
each  analysis 

13.  Develop  computer-assisted  instructions  (CAI)  for  users  of 
the  APM 

Administrative  Items  Development 

14.  Incorporate  on-line  security  and  accounting  procedures 

15.  Provide  mechanisms  for  future  modification  of  the  APM 


Figure  30.  Recommended  Further  Research  on  the  APM 
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B .  Procedures  Development 

4 .  Further  Develop  Procedures  for  APM  Application  to  Measures 

Derivation 


One  of  the  most  difficu)t-to-routinize  segments  of  the  APM  is  chat  in 
which  measurable  attributes  are  determined  and  actual  measures  are  selected. 
Some  progress  has  been  made  during  this  project  in  beginning  to  identify 
specific  procedures  for  measures  derivation.  More  needs  to  be  done,  however, 
especially  if  computer-aiding  is  to  be  used  for  this  segment.  In  particular, 
the  analytic  steps  leading  to  measures  derivation  involve  a  determination  of 
issues  that  imply  a  need  to  measure  particular  attributes.  The  research 
n.ethod  should  include  model  application  with  several  existing  or  developing 
training  systems  to  verify  or  modify  its  component  parts,  its  adequacy,  and  its 
practicality.  The  effort  should  also  produce  improved  and  amplified  pro¬ 
cedures  and  guidelines  for  using  the  model. 

Additional  work  is  needed  to  incorporate  the  measures  of  effec¬ 
tiveness  and  performance  (MOE-MOP)  development  steps  into  the  APM,  identify 
the  relationships  among  MOE-MOP  for  systems-subsystems  levels,  and  identify 
the  relationships  among  the  machine  and  human  components  in  terms  of  con¬ 
tribution  to  MOE-MOP  development  or  specification.  This  subtask  can  build  on 
the  current  B1FVTS  work,  by  continuing  more  sample  or  partial  applications  to 
increasingly  complex  issues  of  military  training,  both  to  verify  and  to  improve 
the  procedural  guidelines.  This  process  is  analogous  to  the  earlier  one  of 
generating  taxonomization  guidelines.  The  procedural  guidelines  would  allow 
for  the  derivation  of  MOE-MOP  specific  to  given  measurement  purposes  and 
provide  for  direct  MOE-MOP  (type)  selection  given  the  defined  measurement 
purpose.  Those  guidelines,  when  better  understood,  will  define  the  algorithm 
for  measures  derivation  that  then  can  be  used  to  improve  the  computer-aided 
process. 


5 .  Further  Develop  Procedures  for  APM  Application  to  System  Design 
Specification 

The  purpose  of  this  research  item  is  analogous  to  that  of  Item  4,  but 
as  applied  to  the  APM  for  specification.  Sample  applications  should  involve  the 
specification  of  several  new  Army  systems.  All  component  parts  of  the 
specification  model  should  be  addressed;  concepts  and  definitions  should  be 
clarified  and  application  guidelines  should  be  developed.  In  addition,  the 
utility  of  the  model  for  assessing  independently  generated  design  requirements 
should  be  evaluated. 

6 .  Develop  Procedures  for  Differentiating  Specifications  and  Measures  in 
Terms  of  Weighting  Factors,  Delay  Tolerance  and  Criticality 

The  practical  aspects  of  model  utilization  suggest  that  there  will  be 
times  when  the  full  range  of  performance  taxa,  attributes,  specifications  or 
measures  cannot  be  applied  in  its  entirety,  and  that  choices  will  have  to  be 
made  to  select  only  the  most  important  ones  for  application.  To  aid  the  user 


in  selecting  those  taxa,  attributes,  specifications  or  measures  which  are  most 
important,  a  procedure  should  be  developed  which  helps  to  rate  or  rank  those 
items  in  accordance  with  such  criteria  as:  criticality  to  mission  performance, 
amenability  to  alternatives  or  corrective  actions,  tolerance  to  postponement  or 
delay  in  application,  or  other  criteria  to  be  determined. 

7.  Modify  Procedures  and  Programs  for  Use  with  a  Computer  Lar^e 

Enough  to  Provide  the  User  Assistance,  Data  Base  and  Processing 

Capacity,  and  Speed  Consistent  with  the  Expanded  Capabilities 

This  task  is  intended  to  produce  a  practical  and  useful  tool  by 
building  upon  the  existing  computer-aided  demonstration  capability.  It  can  be 
accomplished  by  adding  flexibility,  instruction  options,  thoroughness,  and  data 
bases  to  the  rudimentary  computer-based  model.  It  should  also  include  a  test 
of  the  procedures  by  a  skilled,  but  APM-naive,  analyst  to  determine  where 
modifications  arc  required  to  clarify,  simplify  or  otherwise  improve  their  value 
and  ease  of  use. 

8 .  Improve  Procedures  for  Specifying  New  Analytic  Purposes 

This  task  can  build  upon  the  experience  gained  in  conducting 
Task  3.  It  consists  of  analyzing  and  documenting  the  process  by  which 
analytic  purposes  are  specified  and  how  they  subsequently  influence  the 
model's  application  (as  in  Task  3).  It  is  intended  that  the  procedures 
developed  be  incorporated  in  the  computer-aided  model,  to  assist  users  at  the 
appropriate  stage  of  application. 

C.  Applications  and  Application  Aids 

9.  Train  Users  to  Apply  the  APM 

This  task  is  one  in  which  the  previously  developed  APM  procedures 
are  placed  into  operation  with  some  potential  users.  One  possible  agency  to 
aid  in  this  dissemination  function  is  the  U.S.  Army  Training  and  Doctrine 
Command  (TRADOC).  Administrative  contacts,  explorations  and  information 
exchange  with  possible  interested  agencies  should  begin  well  in  advance  of  the 
potential  application  efforts.  The  results  of  these  trials  should  be  used  to 
support  Task  13  in  the  development  of  CAI  resources. 

10.  Apply  the  APM  to  the  Measurement  of  an  Existing  Army  System 

This  task  involves  extending  the  preliminary  BIFVTS  applications  of 
the  APM  to  a  different  measurement  purpose  that  also  can  serve  as  a  validation 
or  verification  exercise.  For  example,  the  model  can  be  applied  in  the  context 
of  the  Training  Effectiveness  Analysis  (TEA)  of  the  BIFVTS  which  was  planned 
by  USAIS/USAIC  to  begin  in  the  4th  Qtr  FY  1982.  Current  developments  ran 
be  extended  to  the  point  where  derivation  of  specific  measure  sets  for  the 
effectiveness  of  the  BIFVTS  subsystem  can  be  generated.  These  measure  sets 
can  then  be  examined  for  feasibility,  validity  and  utility  in  the  context  of  the 
TEA.  It  would  be  necessary  to  review  the  measurement  issues  to  be  addressed 
in  the  TEA  that  are  relevant  to  the  BIFVTS  curriculum,  and  one  or  more  of 
those  issues  should  be  selected  as  the  basis  for  this  task.  The  performance 
taxa  pertaining  to  each  selected  issue  and  the  measurement  derivation 


guideline?  should  be  used  to  generate  a  measures  hierarchy  for  each  selected 
issue.  Differences  between  APM  and  TEA  measures  should  be  examined  so  that 
reasons  can  be  determined  for  those  differences.  Commonalities  should  also  be 
noted  and  their  reasons  understood.  For  practical  reasons,  this  in-depth 
application  and  comparison  should  focus  on  a  narrow  aspect  of  the  BIFVTS, 
such  as  Basic  Gunnery  Learner  Testing.  This  task  would  also  provide  an 
opportunity  to  focus  on  human  operator  contributions  to  performance,  such  as 
motivation,  attitudes,  capabilities,  and  procedures  under  given  environmental, 
administrative  and  other  constraints. 

1 1 .  Apply  the  APM  to  the  Specification  of  a  New  Army  System 

Initial  extension  of  the  APM  to  the  specification  of  system  design 
requiremen's  was  begun  during  the  latter  portion  of  this  contract.  This 
follow-up  task  should  extend  and  further  explore  the  use  of  the  APM  as  a 
tool,  through  application  to  a  new  (or  developing)  system.  A  training  system 
is  recommended  as  one  possibility.  The  purpose  is  to  improve  and  to  verify, 
to  the  extent  possible,  the  capability  of  the  APM  to  serve  this  function.  This 
can  be  done  through  the  "blind"  development  of  training  system  design 
specifications  for  a  ne\.  or  developing  system  to  be  selected  by  an  interested 
Army  office.  A  comparative  assessment  can  be  carried  out  between  design 
specifications  as  developed  by  using  the  APM,  and  similar  specifications  as 
developed  by  other  methods,  such  as  the  Early  Training  Estimation  System 
under  development  by  the  ARI  Fort  Bliss  Field  Unit.  This  task  could  also 
make  use  of  the  APM's  capabilities  both  to  generate  new  design  specifications 
and  to  aid  in  the  diagnostic  assessment  of  previously  generated  design 
specifications.  For  the  same  practical  reasons  as  mentioned  under  Task  9,  this 
comparative  application  should  take  place  in  depth  over  a  narrow  aspect  of  the 
new  training  system. 

12 .  Provide  the  User  with  a  Computer-Generated  Assessment  of  Each 

Analysis 

This  is  a  relatively  small  task  which  will  provide  the  user  with 
feedback  on  how  each  analysis  was  carried  out.  It  will  provide  assessments  in 
terns  of  thoroughness  or  breadth  of  the  analysis,  conformance  with  or 
departures  from  computer  recommendations,  completeness  or  lack  of  same  in 
associating  performance  items  with  final  measures  or  specifications,  and  other 
criteria  by  which  the  computer  can  report  on  the  user's  activities.  It  is 
intended  that  these  assessments  will  prompt  the  user  to  make  changes  or 
improvements  where  necessary  or  to  obtain  some  measure  of  assurance  that  the 
analysis  was  probably  conducted  in  a  satisfactory  manner. 

1 3 .  Develop  Computer-Assisted  Instructions  (CAT)  for  Users  of  the  APM 

This  task  would  provide  the  computer-based  package  with  a  built-in 
training  capability,  using  the  self-pacing  techniques  of  computer-assisted 
instructions  (CAI).  Completion  of  this  task  should  follow  whatever  remaining 
development  effort  is  necessary  to  yield  routinized  (computer-aided) 
procedures  for  application  by  users.  It  should  also  follow  application  tests  of 
those  procedures  by  user  personnel  (sec  Task  9).  This  task  can  make  use  of 
any  handbooks,  guidelines,  workshops  or  other  items  that  were  previously 


developed  to  train  or  assist  users.  The  task  should  also  consider  the 
training/usage  of  the  APM  process  by  various  personnel  types  and  for  both 
possible  purposes  (evaluation  and  specification  of  human-machine  systems). 
CAI  can  insure  that  procedures  for  training  users  of  the  APM  are  as 
programmatic,  proceduralized ,  routinized  and  simplified  as  possible.  Field 
tests  of  these  CAI  procedures  should  aim  to  confirm  their  completeness  and 
accuracy,  their  utility  (based  on  the  acceptance  by  users)  and  their  validitv 
(based  on  the  quality  and  comprehensiveness  of  products). 

D.  Administrative  Items  Development 

14.  Incorporate  On-Line  Security  and  Accounting  Procedures 

This  task  is  designed  to  protect  the  integrity  of  the  model  and  its 
data  bases,  once  it  becomes  accessible  to  a  wide  variety  of  users.  It  requires 
the  incorporation  of  appropriate  access  codes  in  order  to  modify  data  bases  or 
programs.  It  also  provides  for  a  record  of  usage  by  amount,  type,  user 
description  and  other  accounting-type  information  that  can  serve  to  describe 
how  the  APM  is  being  used,  and  to  suggest  how  additional  developmental 
resources  can  be  best  applied. 

15.  Provide  Mechanisms  for  Future  Modification  of  the  APM 


To  help  the  computer-aided  instruction  system  in  carrying  out  its 
functions  of  information  exchange  and  documentation  for  users,  a  technical 
capability  should  be  established  for  maintaining  the  APM,  its  associated  data 
bases,  and  its  various  ;.pplication  routines  and  records.  Users  can  then  be 
helped  with  the  latest  fund  of  knowledge,  techniques  and  experience.  A 
current  directory  of  users  would  allow  for  rapid  distribution  of  application 
documents,  program  modifications,  information  requests,  and  suggestions  to 
improve  the  value  of  this  resource  to  users. 

One  of  the  most  important  sources  of  ideas  to  enhance  the  APM's 
utility  and  value  is  the  user  population  itself.  A  method  should  be  established 
for  permitting  and,  in  fact,  encouraging  users  to  supply  feedback  of 
application  ideas  to  the  central  APM  technical  office,  with  the  assurance  that 
those  ideas  will  be  given  careful  consideration  and  dissemination  in  the  most 
appropriate  manner.  Dissemination  can  employ  a  newsletter,  a  "new 
application"  description,  a  revised  data  base  or  even  a  message  for  CRT 
display  when  users  log  on  to  use  the  model.  In  fact,  one  way  in  which  users 
can  conveniently  communicate  briefly  with  the  APM  central  office  could  be  via 
the  interactive  terminal,  such  as  by  typing  in  messages  or  inquiries  after 
using  the  program  or  by  pausing  during  use  when  a  problem  is  encountered. 
The  specific  mechanisms  for  keeping  the  APM  current  and  useful  should  be 
determined  and  implemented  in  this  task  item. 
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VII.  ANNOTATED  GLOSSARY  OF  ANALYTIC  PROCESS  MODEL 

TERMS  AND  CONCEPTS 


Numerous  concepts,  procedures,  techniques  and  data  bases  are  presented  in 
this  report.  Of  necessity,  these  are  expressed  in  technical  terminology,  much 
of  which  is  specific  to  training  applications.  Understanding  and  application  of 
these  concepts,  data  bases,  etc.,  requires  understanding  of  the  terminology. 
This  glossary  defines  and  clarifies  the  terms  used,  in  an  attempt  to  foster  that 
understanding . 

The  definitions  and  clarifications  presented  herein  are  drawn  principally  from 
the  following  sources: 
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GLOSSARY 


ACHIEVEMENT. 

See:  LEARNING  ACHIEVEMENT 

ACTION  VERB.  The  action  verb  is  the  behavior  indicator  of  a  performance 
objective. 

In  a  statement  of  a  performance  objective,  the  action  verb  discloses  the 
observable  behavior  that  the  learner  is  required  to  manifest. 

See  also:  PERFORMANCE  OBJECTIVE 
CAPABILITY  VERB 

ADULT  LEARNING  CLIMATE.  The  adult  learning  climate  is  that  set  of 

environmental,  procedural,  and  interpersonal  features  that,  typically, 
facilitates  learning  by  adults. 

Features  of  the  adult  learning  climate: 

1.  Learners  are  treated  with  respect. 

2.  Learners  participate  in  the  learning  planning  process,  helping  to 
translate  diagnosed  needs  into  specific  performance  objectives  and 
learning  activities. 

3.  Learners  participate  in  the  diagnosis  of  their  own  needs. 

4.  Learners'  comments,  questions,  discussions  are  encouraged  and 
welcome. 

5.  Learners  share  in  conducting  learning  experiences. 

6.  Learners  have  opportunities  for  self-evaluation. 

7.  Emphasis  is  placed  on  the  practical  applications  of  the  learning. 

8.  Experiential  techniques  are  used  to  the  fullest  possible  extent  (e.g., 
group  discussion,  simulations,  case  studies,  role-playing,  etc.) 

Synonym:  ADULT  LEARNING  ENVIRONMENT 

See  also:  EXTERNAL  CONDITIONS  OF  LEARNING 
INDIVIDUALIZED  INSTRUCTION 

ANALYSIS.  The  first  phase  of  instructional  system  development. 

Comprises  the  early  stages  of  curriculum  design,  such  as  the  analysis  of 
needs,  goals  and  objectives,  and  the  organization  of  course  units. 

Entails  procedures  for  defining  the  job  to  be  trained;  dividing  the  job 
down  into  statements  of  tasks;  and  selecting  the  tasks  to  be  trained. 

Further  extends  to  constructing  job  performance  measures  for  tasks 
selected  for  training,  and  to  deciding  whether  the  tasks  should  be  trained 
in  schools,  on  the  job,  or  elsewhere. 


Synonym:  FRONT  END  ANALYSIS 

See  also:  TRAINING  NEEDS  ASSESSMENT 
JOB  ANALYSIS 

ANALYSIS  OF  EXISTING  COURSES.  In  instructional  system  development, 

analysis  of  existing  courses  is  undertaken  to  avoid  needless  duplication  of 
effort,  by  permitting  existing  courseware  to  be  adapted  to  new  training 
needs. 

Factors  to  consider  in  the  analysis  of  existing  courses: 

1.  The  acceptability  of  the  job  analysis  on  which  the  existing  course 
was  based. 

2.  The  appropriateness  of  the  criteria  that  were  used  to  select  tasks  to 
be  trained  through  the  existing  course. 

3.  The  validity  of  the  job  performance  measures  used  in  the  existing 
course. 

4.  Validation  of  the  existing  course. 

ANALYSIS  OF  PERFORMANCE  OBJECTIVES. 

See:  TASK  ANALYSIS 

ANALYTIC  PROCESS  MODEL.  The  Analytic  Process  Model  is  a  conceptual 

framework  for  systematizing  the  design  and  evaluation  of  human-machine 
systems. 

Stages  of  the  Analytic  Process  Model: 

1.  Establish  the  Design/Evaluation  context  by  identifying  the  system's 
work  requirements.  (This  is  the  Systems  Taxonomy  Model  stage.) 

2.  Establish  the  focus  of  the  Design/Evaluation  in  terms  of  necessary 
system  attributes. 

3.  Implement  the  application  details  to  Design/Evaluate  the  attributes. 

4.  Interpret  the  outcome  of  the  Design/Evaluation,  and  develop 
conclusions  and  recommendations. 

Synonym:  APM 

See  also:  SYSTEMS  TAXONOMY  MODEL 
DESIGN 
EVALUATION 


APM. 


See:  ANALYTIC  PROCESS  MODEL 


ASPECTS  OF  PERFORMANCE.  The  aspects  of  human-machine  system 

performance  form  one  dimension  for  organizing  the  system's  work 
requirements.  The  aspects  of  performance  address  the  work  itself  and 
the  circumstances  under  which  the  work  is  required. 

The  aspects  of  performance  include  five  mutually  exclusive  categories: 

1 .  Performance  Potentialities 

(inherent  capabilities) 

2.  Performance  Processes 

(activities,  methods,  techniques) 

3.  Performance  Products 

(goods,  services,  other  deliverables) 

4.  Performance  Environment 

(natural  impediments  to  the  work) 

5.  Performance  Constraints 

(man-made,  artificial  impediments) 

The  aspects  of  performance  form  one  dimension  of  the  Systems  Taxonomy 
Model.  They  interact  with  another  dimension,  viz.,  levels  of  system 
description,  to  form  fifteen  cells  or  sub-taxonomies  within  which  any 
system's  work  requirements  can  be  exhaustively  identified  and  organized. 

See  also:  SYSTEMS  TAXONOMY  MODEL 

LEVELS  OF  SYSTEM  DESCRIPTION 
PERFORMANCE  POTENTIALITIES 
PERFORMANCE  PROCESSES 
PERFORMANCE  PRODUCTS 
PERFORMANCE  ENVIRONMENT 
PERFORMANCE  CONSTRAINTS 

ATTITUDINAL  DOMAIN  OBJECTIVES.  Performance  objectives  belonging  to  the 
attitudinal  domain  require  that  the  learner  demonstrate  that  he  or  she 
chooses  some  prescribed  course  of  personal  action,  or  behaves  in 
accordance  with  a  prescribed  set  of  values. 

See  also:  PERFORMANCE  OBJECTIVES 

LEARNING  DOMAIN  TAXONOMY  MODELS 
INFORMATION  DOMAIN  OBJECTIVES 
MENTAL  SKILLS  DOMAIN  OBJECTIVES 
PHYSICAL  SKILLS  DOMAIN  OBJECTIVES 

ATTRIBUTE.  An  attribute  of  a  human-machine  system  is  a  concrete, 

observable  feature  of  the  system  that  bears  directly  on  the  work  that  the 
system  can  and  does  perform. 

Attributes  are  linked  to  specific  taxa  of  performance.  For  each 
performance  taxon,  there  exists  a  set  of  system  attributes  that 
collectively  determine  whether  the  system  satisfies  that  taxon. 
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A  system  attribute  is  something  that  a  system  designer  can  and  must 
control,  through  design  decisions,  to  insure  that  the  S/Stem  will  satisfy 
the  performance  taxon  to  which  the  attribute  is  linked.  An  attribute  also 
is  something  that  an  evaluator  can  and  must  measure  to  determine  whether 
the  performance  taxon  is  met. 

See  also:  TAXON 

MEASURE 

DESIGN  SPECIFICATION 
BEHAVIORAL  OBJECTIVE. 

See:  PERFORMANCE  OBJECTIVE 

CAPABILITY  VERB.  The  capability  verb  of  a  performance  objective  statement 
indicates  the  particular  domain  of  learning  to  which  the  objective  belongs. 

See  also:  PERFORMANCE  OBJECTIVE 
DOMAINS  OF  LEARNING 
ACTION  VERB 

CHARACTERISTICS-LEVEL  TAX  A.  Characteristics-level  taxa  are  statements  of 
how  a  specific  purpose  behind  a  system's  work  is  required  to  be 
achieved.  Characteristics-level  taxa  define  the  particular  ingredients, 
methods  and  milestones  that  the  system  is  to  include  in  its  work  in  order 
to  satisfy  its  reasons  for  working. 

A  Characteristics-level  taxon  is  linked  to  or  descends  from  a  particular 
Functional  Purooses-level  taxon.  The  Characteristics-level  taxon  defines 
a  detailed  capability  ingredient,  or  procedural  step,  or  intermediate  . 
output  necessary  to  achieve  a  particular  purpose  behind  some  basic 
potentiality,  process  or  product.  Characteristics-level  taxa  also  include 
definitions  of  the  relevant  environmental  and  constraining  factors  that  may 
interfere  with  the  system's  ability  to  provide  these  capability  ingredients, 
carry  out  these  procedural  steps,  or  deliver  these  intermediate  outputs. 

Characteristics-level  taxa  consist  of  the  system's  performance 
potentialities,  processes,  products,  environment  and  constraints  viewed  at 
the  bottom-most  level  of  system  description.  It  is  a  view  of  work 
requirements  as  things  that  are  to  be  achieved  in  certain  ways. 

See  also:  TAXON 

OBJECTIVES-LEVEL  TAXA 
FUNCTIONAL  PURPOSES-LEVEL  TAXA 
LEVELS  CF  SYSTEM  DESCRIPTION 
ASPECTS  OF  PERFORMANCE 

COGNITWE  STRATEGIES.  A  learner's  cognitive  strategies  are  the  internally 
organized  mental  skills  by  means  of  which  the  learner  exercises  control 
over  his  or  her  own  processes  of  attending,  learning,  remembering,  and 
thinking . 

See  also:  LEARNING  DOMAIN  TAXONOMY  MODELS 
INTERNAL  CONDITIONS  OF  LEARNING 
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COLLATERAL  SYSTEM.  A  collateral  system  of  a  particular  human-machine 
system  of  interest  is,  itself,  a  human-machine  system  that  interacts 
directly  with  the  system  of  interest  so  that  it  affects  the  work  required 
of  the  system  of  interest. 

A  collateral  system  may  work  either  cooperatively  or  in  competition  with 
the  system  of  interest. 

Collateral  systems  are  members  of  the  system  hierarchy  of  the  system  of 
interest . 

See  also:  HUMAN-MACHINE  SYSTEM 
SYSTEM  HIERARCHY 

COMMAND  SUBSYSTEM.  The  Command  Subsystem  is  one  of  the  six  generic, 
functionally  oriented  subsystems  of  any  training  system.  Command 
provides  administrative  control  over  the  training  system  by  assessing 
training  needs;  allocating  resources;  recruiting  training  system  personnel; 
and  monitoring,  evaluating  and  recording  learner  and  system  performance. 

Command  contributes  to  the  learning-helping  function  by  constructing  and 
managing  the  system  in  which  the  learning  activities  can  take  place. 

The  principal  operator  or  staff  member  of  the  Command  Subsystem  is  the 
training  administrator. 

See  also:  TRAINING  SYSTEM 
LEARNING 
LEARNING-HELPING 

CONDITIONS  OF  LEARNING.  The  conditions  of  learning  are  those  features  or 
qualities  of  the  training  system,  including  the  personnel,  equipment  and 
procedures,  that  affect  how  well  a  particular  learner  learns. 

See  also:  INTERNAL  CONDITIONS  OF  LEARNING 
EXTERNAL  CONDITIONS  OF  LEARNING 

CRITERION-REFERENCED  TEST. 

See:  OBJECTIVE-REFERENCED  TEST 

CUE.  A  cue  is  a  subtle  or  indirect  form  of  guidance,  given  to  help  a  learner 
recall  and  apply  some  particular  fact,  procedure,  word,  phrase,  etc., 
without  actually  supplying  the  fact,  procedure,  etc. 

See  also:  GENERAL  LEARNING  GUIDELINES 
PROMPTING 

DECISION  FLOW  CHARTING.  Decision  flow  charting  is  a  procedure  for 

identifying  and  assessing  the  decisions  required  in  a  task,  by  means  of 
reduction  to  a  series  of  binary  choices. 

See  also:  INFORMATION  PROCESSING  ANALYSIS 
DECISION  LEVEL  ANALYSIS 
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DECISION  LEVEL  ANALYSIS.  Decision  level  analysis  is  applied  to  the 

decisions  required  in  a  task  to  establish  a  rating  of  the  time,  complexity 
and  nature  of  each  decision. 

See  also:  INFORMATION  PROCESSING  ANALYSIS 
DECISION  FLOW  CHARTING 

DELIVERY  SUBSYSTEM.  The  Delivery  Subsystem  is  one  of  the  six  generic, 

functionally  oriented  subsystems  of  any  training  system.  Delivery  is  the 
most  important  training  subsystem,  in  that  the  primary  training  function 
(learning)  takes  place  there.  In  the  Delivery  Subsystem,  the  prescribed 
sensory  learning  activities  are  carried  out  and  the  learner  achieves  the 
intended  behavioral  effects. 

The  principal  operator  of  the  Delivery  Subsystem  is  the  learner.  The 
learner  is  assisted  in  Delivery  by  all  other  training  system  personnel, 
who  play  various  learning-helping  roles. 

See  also:  TRAINING  SYSTEM 
LEARNER 
LEARNING 
LEARNING-HELPING 

DEPENDENT  OBJECTIVE.  One  performance  or  enabling  objective  is  dependent 
upon  another  if  it  cannot  be  achieved  unless  the  other  first  is  achieved. 

If  one  objective  is  dependent  upon  another,  it  must  be  pursued  after  the 
other,  during  training. 

See  also:  INDEPENDENT  OBJECTIVES 
SUPPORTIVE  OBJECTIVES 
PERFORMANCE  OBJECTIVE 
SEQUENCING  OBJECTIVES 

DESIGN.  Usage  No.  1  (Principal  Definition):  One  of  two  applications  of  the 
Analytic  Process  Model.  Design  of  a  human-machine  system  is  the  process 
of  selecting  and  securing  all  of  the  attributes  that  the  system  must 
possess  in  order  to  satisfy  its  work  requirements. 

Usage  No.  2  (Secondary  Definition):  One  of  six  generic  subsystems  of 
any  training  system. 

Synonym:  DESIGN  SUBSYSTEM 

See  also:  ANALYTIC  PROCESS  MODEL 
ATTRIBUTE 

DESIGN  SPECIFICATION 
EVALUATION 

DESIGN  SPECIFICATION.  A  design  specification  is  a  plan  for  insuring  that  a 
system  will  have  some  particular  required  attribute. 
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A  design  specification  describes  how  the  system  will  be  built,  maintained 
and  operated  so  that  a  specific  attribute  needed  to  satisfy  a  particular 
performance  requirement  or  taxon  will  be  present. 

See  also:  ATTRIBUTE 
TAXON 
MEASURE 

DESIGN  SUBSYSTEM.  The  Design  Subsystem  is  one  of  the  six  generic, 

functionally  oriented  subsystems  of  any  training  system.  Design  carries 
out  the  development  of  the  curriculum/program  of  instruction.  In  so 
doing,  Design  plans  the  instructional  events,  selects  training  technology, 
assembles  content  material,  and  defines  facilitator  and  learner 
requirements. 

The  Design  Subsystem  is  the  portion  of  the  training  system  in  which  the 
Instructional  System  Development  (ISD)  model  is  applied. 

Design  Subsystem  contributes  to  the  learning-helping  function  by 
identifying  the  specific  behavioral  effects  needed  by  the  learners  and  by 
planning  the  sensory  activities  that  will  lead  to  those  effects. 

The  principal  operator  of  the  Design  Subsystem  is  the  curriculum 
developer. 

See  also:  TRAINING  SYSTEM 

INSTRUCTIONAL  SYSTEM  DESIGN 

LEARNING 

LEARNING-HELPING 

DEVELOPMENT  OF  OBJECTIVES.  Development  of  objectives  is  the  process  by 
which  performance  objectives,  enabling  objectives  and  prerequisites  for 
training  are  identified. 

Steps  in  the  development  of  objectives: 

1.  Identify  performance  objectives  by  translating  job  performance 
measures  into  training-oriented  terms. 

2.  Analyze  the  performance  objectives  (tasks)  to  identify  constituent 
prerequisite  or  supportive  achievements. 

3.  Continue  the  analysis  until  a  level  of  prerequisites  is  reached  which 
all  learners  have  achieved. 

Product  of  the  development  of  objectives: 

A  list  of  performance  objectives  and  prerequisites  analyzed  to  the 
level  where  it  c~n  be  assumed  that  the  individuals  to  be  trained  are 
prepared  to  begin. 
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See  also:  TASK  ANALYSIS 

PERFORMANCE  OBJECTIVE 
ENABLING  OBJECTIVE 
ESSENTIAL  PREREQUISITE 
SUPPORTIVE  PREREQUISITE 

DOMAINS  OF  LEARNING.  The  domains  of  learning  are  categories  into  which 
performance  objectives,  enabling  objectives,  and  prerequisites  may  be 
grouped,  in  accordance  with  whether  the  prescribed  action/behavior  is 
mental,  physical,  or  attitudinal. 

See  also:  LEARNING  DOMAIN  TAXONOMY  MODELS 

EMPLACEMENT  SUBSYSTEM.  The  Emplacement  Subsystem  is  one  of  the  six 
generic,  functionally  oriented  subsystems  of  any  training  system. 
Emplacement  takes  the  plans  developed  by  Design  Subsystem  and 
constructs,  acquires  or  assembles  all  materials,  equipment,  installations, 
supplies,  etc.,  needed  to  implement  those  plans. 

Emplacement  contributes  to  the  learning-helping  function  by  providing  the 
tools  needed  to  carry  out  the  sensory  learning  activities. 

The  principal  operator  of  the  Emplacement  Subsystem  is  the  facilities 
developer. 

See  also:  TRAINING  SYSTEM 
DESIGN  SUBSYSTEM 
LEARNING 
LEARNING-HELPING 

ENABLING  OBJECTIVE.  Ar  enabling  objective  is  a  learned  capability  that 

facilitates  the  learning  of  a  larger  performance  objective.  An  enabling 
objective  can  be  conceived  as  an  intermediate  achievement,  or  milestone, 
that  must  be  or  can  be  accomplished  to  progress  toward  the  final 
achievement  represented  by  the  performance  objective. 

See  also:  PERFORMANCE  OBJECTIVE 
ESSENTIAL  PREREQUISITE 
SUPPORTIVE  PREREQUISITE 
LEVELS  OF  OBJECTIVES 
LESSON  PLAN 

ENABLING  SUBSYSTEM.  The  Enabling  Subsystem  is  one  of  the  six  generic, 
functionally  oriented  subsystems  of  any  training  system.  Enabling  is, 
itself,  a  self-contained  training  system:  it  carries  out  learning  and 
learning-helping  activities  to  enable  facilitators  to  (1)  become  familiar  with 
the  plans,  training  content,  equipment  and  facilities  provided  by  the 
Design  and  Emplacement  Subsystems;  (2)  become  qualified  and  prepared 
to  teach  the  curriculum  effectively;  and  (3)  tailor  the  curriculum  to  the 
specific  needs  of  a  particular  class  or  team  of  learners. 
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Within  the  total  training  system,  Enabling  contributes  to  the 
learning -helping  function  by  providing  personnel  qualified  to  assist  the 
learners  directly  by  presenting  information,  demonstrating  techniques, 
coaching  learners'  efforts,  evaluating  learners'  performance,  and 
correcting  learners'  deficiencies.  Enabling  also  contributes  to 
learning-helping  by  fine-tuning  a  general  program  of  instruction  to 
specific  learning  needs  of  specific  learners. 

The  principal  operator  of  the  Enabling  Subsystem  is  the  facilitator. 

Common  synonyms  for  Enabling  Subsystem  are  "Instructor  Preparation" 
and  "Training-the-Trainer. " 

See  also:  TRAINING  SYSTEM 
DESIGN  SUBSYSTEM 
EMPLACEMENT  SUBSYSTEM 
FACILITATOR 
LEARNING 
LEARNING-HELPING 

ENTRY  TEST.  An  entry  test  is  a  test  of  a  prerequisite  for  training. 

Entry  tests  provide  a  basis  for  accepting  or  rejecting  candidate  learners, 
and  for  determining  whether  particular  learners  require  remedial  training. 

See  also:  OBJECTIVE-REFERENCED  TEST 
PRETEST 
POSTTEST 

ERROR  DATA.  Error  data  consist  of  item  analyses  of  pilot  test  learners'  test 
responses.  The  analyses  disclose  achievement  failures,  which  in  turn 
point  to  needed  improvements  in  the  instruction. 

See  also:  FEEDBACK 

ESSENTIAL  PREREQUISITE.  An  essential  prerequisite  for  a  particular 

performance  objective  or  enabling  objective  is  a  performance  capability 
that  actually  is  incorporated  within  the  objective  in  question. 

The  objective  cannot  be  achieved  unless  the  essential  prerequisite  is 
satisfied. 

See  also:  PERFORMANCE  OBJECTIVE 
ENABLING  OBJECTIVE 
DEPENDENT  OBJECTIVE 
DEVELOPMENT  OF  OBJECTIVES 
SUPPORTIVE  PREREQUISITE 

EVALUATION.  Evaluation  is  one  of  two  applications  of  the  Analytic  Process 
Model.  Evaluation  of  a  human-machine  system  is  the  process  of 
determining  whether  the  system  satisfies  all  of  its  work  requirements. 

Synonym:  MEASUREMENT 
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See  also:  ANALYTIC  PROCESS  MODEL 
MEASURE 
DESIGN 

EXTERNAL  CONDITIONS  OF  LEARNING.  A  learner's  external  conditions  of 
learning  for  a  particular  performance  or  enabling  objective  are  those 
features  or  qualities  supplied  by  the  facilitator,  the  learning  materials  and 
media,  the  facilities,  etc.,  that  affect  the  particular  learner's  ability  to 
achieve  the  particular  objective. 

See  also:  TACTICS  OF  INSTRUCTION 
INSTRUCTIONAL  FEATURES 
CONDITIONS  OF  LEARNING 
INTERNAL  CONDITIONS  OF  LEARNING 
ADULT  LEARNING  CLIMATE 

FACILITATOR.  A  facilitator  is  a  key  learning-helper  in  a  training  system, 
and  is  the  principal  operator  of  the  training  Enabling  Subsystem. 

The  facilitator  interacts  directly  with  the  learner  to  aid  the  learner  in 
carrying  out  planned  sensory  activities  to  achieve  desired  behavioral 
effects. 

Common  synonyms  for  "facilitator"  include  "teacher,"  "instructor,"  and 
"trainer. " 

See  also:  LEARNING-HELPING 
TRAINING  SYSTEM 
ENABLING  SUBSYSTEM 
DELIVERY  SUBSYSTEM 
LEARNER 

FEEDBACK.  In  course  design,  feedback  is  the  process  of  gathering 
information  from  learners  as  the  basis  for  improving  the  training. 

See  also:  FORMATIVE  EVALUATION 
PILOT  TESTING 
ERROF.  JV.TA 


FIELD  TRIAL. 

See:  PILOT  TESTING 

FORMATIVE  EVALUATION.  Formative  evaluation  is  the  tryout  and  revision  of 
a  training  system  prior  to  its  full-scale  operational  delivery.  Formative 
evaluation  entails  delivery  of  the  instruction  to  representatives  of  the 
intended  learner  population,  collection  of  useful  data  from  those 
representatives,  analysis  and  interpretation  of  those  data,  and  revision  of 
the  instruction  based  upon  the  findings. 

The  word  "formative"  implies  that  the  evaluation  occurs  while  the 
instructional  program  is  still  in  development.  Formative  evaluation 
provides  an  opportunity  to  correct  mistakes  made  in  the  instructional 
design  process  before  the  delivery  process  fully  matures. 
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See  also:  PILOT  TESTING 
FEEDBACK 

SUMMATIVE  EVALUATION 
FRONT  END  ANALYSIS. 

See:  ANALYSIS 

FUNCTIONAL  PURPOSES-LEVEL  TAXA.  Functional  Purposes-level  taxa  are 

statements  of  why  specific  performance  is  required  of  a  system,  without 
reference  to  how  the  requirement  is  to  be  achieved. 

A  Functional  Purposes-level  taxon  is  linked  to  or  descends  from  a 
particular  Objectives -level  taxon.  The  Functional  Purposes-level  taxon 
defines  a  specific  reason  why  the  system  is  expected  to  possess  a 
particular  basic  capability,  or  carry  out  a  basic  activity,  or  deliver  a 
basic  output. 

Functional  Purposes-level  taxa  also  include  definitions  of  the  relevant 
environmental  and  constraining  factors  that  may  impede  satisfaction  of  the 
reasons  behind  the  system's  work  requirements. 

Functional  Purposes-level  taxa  consist  of  the  system's  Performance 
Potentialities,  Processes,  Products,  Environment  and  Constraints  viewed 
at  the  middle  level  of  system  description.  It  is  a  view  of  work 
requirements  as  things  that  exist  for  specific  purposes,  the  achievement 
of  which  merits  study. 

See  also:  TAXON 

OBJECTIVES-LEVE1,  TAXA 
CHARACTERISTICS-LEVEL  TAXA 
LEVELS  OF  SYSTEM  DESCRIPTION 
ASPECTS  OF  PERFORMANCE 

GENERAL  LEARNING  GUIDELINES.  The  general  learning  guidelines  consist  of 
four  requirements  deemed  essential  for  helping  learners  to  learn: 

1.  Inform  the  learner  of  the  objective. 

2.  Provide  for  active  practice  by  the  learner. 

3.  Provide  guidance  and  prompting  to  the  learner. 

4.  Provide  feedback  to  the  learner  concerning  his/her  performance. 

See  also:  INSTRUCTIONAL  EVENTS 
PROMPTING 
CUE 

GOAL.  A  goal  is  a  statement  of  what  "ought  to  be"  with  respect  to  learners' 
or  job  performers'  achievements. 

In  the  training  context,  goals  usually  are  broad  statements  of  the  ultimate 
intended  outcomes  of  educational  enterprises. 

See  also:  NEED 

NEEDS  ASSESSMENT 
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HUMAN-MACHINE  SYSTEM.  "Human-machine  system"  is  the  general  entity  of 
interest  to  the  Analytic  Process  Model.  It  is  a  collection  of  people, 
equipment  and  procedures  that  work  together  to  accomplish  specified 
functions. 

INDEPENDENT  OBJECTIVES.  Two  performance  objectives,  or  two  enabling 

objectives,  are  independent  if  the  learning /achievement  of  one  of  them  has 
no  effect  on  the  learning /achievement  of  the  other. 

If  two  objectives  are  independent,  the  sequence  in  which  they  are 
pursued  during  training  is  irrelevant. 

See  also:  DEPENDENT  OBJECTIVE 
SUPPORTIVE  OBJECTIVES 
PERFORMANCE  OBJECTIVE 
SEQUENCING  OBJECTIVES 

INDIVIDUALIZED  INSTRUCTION.  Individualized  instruction  is  a  training 
delivery  subsystem  that  permits: 

1.  Learners  to  choose  their  own  individual  performance  objectives,  or 

2.  Alternative  media  to  be  selected  by  individual  learners  to  pursue 
objectives,  or 

3.  Learners  to  progress  at  their  own  rates. 

Individualized  instruction  is  training  in  which  differences  in  individuals' 
needs,  learning  rates,  attention  spans,  learning  styles,  levels  of 
competence,  etc.,  are  reflected  in  the  learning  activities  experienced  by 
individual  learners.  Individualized  instruction  is  a  basic  requirement  for 
all  adult  training. 

An  individualized  instructional  system  should  make  it  possible  for 
individual  learners  to  progress  at  their  own  rates  of  learning  and  to 
continue  the  instructional  process  until  mastery.  The  instructional  system 
must  obtain  reliable  and  timely  information  on  an  individual  learner's 
learning  progress  and  make  adaptations  appropriate  to  the  individual 
learner. 

See  also:  INTERNAL  CONDITIONS  OF  LEARNING 
ADULT  LEARNING  CLIMATE 
LEARNING  STYLE 

INFORMATION  DOMAIN  OBJECTIVES.  Performance  objectives  belonging  to  the 
information  domain  require  that  the  learner  demonstrate  the  ability  to 
recall  bodies  of  knowledge. 
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See  also:  PERFORMANCE  OBJECTIVES 

LEARNING  DOMAIN  TAXONOMY  MODELS  (cont'd) 

MENTAL  SKILLS  DOMAIN  OBJECTIVES 
PHYSICAL  SKILLS  DOMAIN  OBJECTIVES 
ATTITUDINAL  DOMAIN  OBJECTIVES 

INFORMATION  PROCESSING  ANALYSIS.  A  component  of  Task  Analysis, 

information  processing  analysis  is  applied  to  a  performance  objective  to 
reveal  the  sequence  of  mental  operations  it  requires. 

Information  processing  analysis  is  premised  on  the  notion  that  complex 
human  performance  can  be  conceived  as  composed  of  simpler  parts.  The 
fundamental  rule  for  information  processing  analysis  is:  carry  the 
analysis  to  the  point  where  you  are  convinced  that  the  revealed 
operations  are  simple. 

Information  processing  analysis  often  is  depicted  via  flowcharting;  the 
flowchart  nodes  consist  of  information  inputs,  mental  actions/computations, 
and  decisions. 

See  also:  TASK  ANALYSIS 

TASK  CLASSIFICATION 
LEARNING  TASK  ANALYSIS 

INSTRUCTIONAL  EVENTS.  The  instructional  events  are  the  distinct  steps  in 
a  lesson  through  which  the  learner,  facilitator  and  material  interact  to 
progress  toward  the  lesson  objective. 

The  instructional  events  include: 

1.  Gaining  learner's  attention. 

2.  Informing  learner  of  the  objective. 

3.  Stimulating  recall  of  prerequisites. 

4.  Presenting  stimulus  material. 

5.  Providing  learning  guidance. 

6.  Eliciting  the  performance. 

7.  Providing  feedback  about  performance  correctness. 

8.  Assessing  the  performance. 

9.  Enhancing  retention  and  transfer. 

See  also:  LESSON  PLAN 

STRATEGY  OF  INSTRUCTION 
TACTICS  OF  INSTRUCTION 

INSTRUCTIONAL  FEATURES,  Instructional  features  are  those  components  of  a 
lesson  that  are  distinct  from  the  instructional  content.  The  instructional 
features  include  the  statements  of  performance  objectives  and  enabling 
objectives,  self-tests,  practice  exercises,  discussion  topics,  etc. 

INSTRUCTIONAL  SETTING.  The  instructional  setting  for  a  particular  task  or 
performance  objective  is  the  context,  situation  or  type  of  environment  in 
which  training  will  take  place  for  that  task/objective. 


Military  training  systems  typically  use  one  or  more  of  these  settings: 

1.  Job  Performance  Aid  (JPA) 

2.  Self-Teaching  Exportable  Package  (STEP) 

3.  Formal  On-the-Job  Training  (FOJT)‘ 

4.  Installation  Support  School  (ISS) 

5.  Resident  School  (RS) 

INSTRUCTIONAL  SYSTEM  DESIGN.  Instructional  system  design  is  the  broad 
application  of  the  "systems  approach"  to  training.  It  entails  the  actual 
preparation  of  instruction.  It  involves  determination  of  how  the  learners 
will  be  managed,  the  kinds  of  learning  experiences  they  will  have,  the 
activities  in  which  they  will  engage,  and  the  form  and  content  of  the 
instructional  delivery  system. 

Steps  in  instructional  system  design: 

1.  Determine  training  needs. 

2.  Organize  the  course  and  the  units  of  instruction. 

3.  Write  performance  objectives. 

4.  Analyze  the  objectives. 

5.  Prepare  assessments  of  learner  performance. 

6.  Sequence  instruction  within  each  objective. 

7.  List  instructional  events  for  enabling  objectives. 

8.  Select  stimulus  types. 

9.  Select  instructional  media. 

10.  Select  conditions  of  learning. 

11.  Write  prescriptions/plans. 

12.  Develop  materials. 

13.  Conduct  formative  evaluations. 

14.  Train  pilot  test  facilitators. 

15.  Conduct  pilot  test. 

16.  Make  final  revisions. 

17.  Conduct  summative  evaluations. 

Key  products  of  instructional  system  design: 

1.  A  performance  objective  corresponding  to  each  task  to  be  trained. 

2.  Test  items  to  measure  each  performance  objective. 

3.  Test  of  entry-level  behaviors  to  validate  original  assumptions. 

4.  Sequencing  of  instruction  for  all  dependent  tasks/enabling 
objectives. 

Synonym: INSTRUCTIONAL  SYSTEMS  DEVELOPMENT 

INTERNAL  CONDITIONS  OF  LEARNING.  A  learner's  internal  conditions  of 
learning  for  a  particular  performance  or  enabling  objective  are  those 
features  or  qualities  within  the  learner,  himself  or  herself,  that  affect  the 
learner's  ability  to  achieve  that  objective. 

A  learner's  internal  conditions  of  learning  include  his  or  her  motivation 
with  respect  to  the  objective  of  interest,  recall  of  prior  learning. 
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satisfaction  of  prerequisites,  cognitive  strategies,  and  any  other 
tendencies,  practices,  memories,  etc.,  of  that  particular  learner  that 
affects  his  or  her  achievement  of  that  particular  objective. 

Individual  learners'  motivation  levels  may  be  affected  by: 

1.  The  learning  content. 

2.  The  information  format. 

3.  The  sequencing  of  instruction. 

4.  Personal  attitudes  and  values. 

Individual  learners  vary  in: 

1.  The  learning  sources  they  prefer  and  from  which  they  learn  best. 

2.  The  learning  conditions  under  which  they  are  most  comfortable  and 

effective. 

3.  Some  aspects  of  information  processing  style. 

4.  Other  personal  abilities  and  characteristics. 

See  also:  CONDITIONS  OF  LEARNING 

PRESENT  CONDITION  OF  LEARNING 
COGNITIVE  STRATEGIES 
EXTERNAL  CONDITIONS  OF  LEARNING 

JOB  ANALYSIS.  The  process  of  describing  all  of  the  performance  elements,  or 
tasks,  that  make  up  a  job. 

Entails  finding  out  exactly  what  job  performers  do  when  they  do  the  job, 
the  order  in  which  they  do  it,  and  the  level  of  skill  or  performance 
deemed  adequate  for  the  job. 

Job  analytic  procedures: 

1 .  Initial  development  of  a  tentative  task  list  from  existing  written 
doctrine,  expert  panel,  analysis  of  similar  jobs,  and/or  joint  effort 
with  developers  of  a  new  human-machine  system  for  which  the  job  is 
required. 

2.  Authentication  of  the  tentative  task  list  using  actual  job  incumbents. 

3.  Validation  of  the  task  list  using  other  job  incumbents. 

4.  Collection  of  task  conditions;  cues  that  initiate  and  guide  the  task 
performance;  and  standards  representing  adequate  task  performance. 

Job  analytic  products: 

1.  A  validated  list  of  the  tasks  comprising  the  job  under  study. 

2.  Itemization  of  the  conditions  under  which  each  task  is  performed. 

3.  Specification  of  the  clues  for  initiating  the  tasks. 
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4.  Specification  of  the  standards  of  performance  of  the  tasks. 

5.  Listing  of  the  elements  of  each  task. 

JOB  PERFORMANCE  MEASURE.  A  job  performance  measure  is  a  test  of  the 
adequacy  of  a  job  incumbent's  work  under  actual  job  application 
conditions.  It  is  an  on-the-job  assessment  of  how  well  the  job  is  done. 

A  job  performance  measure  must  be  established  for  each  task  that  is 
selected  for  training.  The  job  performance  measure  should  be  the  best 
available  indicator  of  how  well  the  task  is  performed. 

The  job  performance  measure  must  be  validated  under  field  conditions. 
To  be  valid,  when  it  is  administered  to  job  incumbents,  it  must 
distinguish  those  who  can  perform  the  task  satisfactorily  from  those  who 
can't. 

Job  performance  measures  form  the  basis  for  developing  performance 
objectives. 

See  also:  JOB  ANALYSIS 
TASK  ANALYSIS 
PERFORMANCE  OBJECTIVE 
PREDICTIVE  VALIDITY  OF  TRAINING 

LEARNER.  A  learner  is  the  key  operator  of  a  training  system,  and  the 

principal  operator  of  the  training  Delivery  Subsystem.  The  learner  is  the 
person  who  carries  out  and  experiences  planned  sensory  activities  to 
achieve  intended  behavioral  effects. 

Common  synonyms  for  "learner"  include  "student"  and  "trainee." 

See  also:  LEARNING 

TRAINING  SYSTEM 
DELIVERY  SUBSYSTEM 
FACILITATOR 

LEARNING.  Learning  is  any  activity  involving  the  senses  that  affects 
behavior. 

Learning  is  the  principal  function  of  a  training  system. 

In  a  training  system,  desired  human  behavioral  effects  are  achieved 
through  the  conduct  of  planned  human  sensory  activities. 

See  also:  LEARNER 

LEARNING-HELPING 
TRAINING  SYSTEM 

LEARNING  ACHIEVEMENT.  A  learner's  learning  achievement  is  how  well  he  or 
she  performs  specific  performance  objectives  of  interest. 
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Learning  achievement  is  performance  evaluated  with  regard  to  its 
adequacy . 

Synonym:  ACHIEVEMENT 

LEARNING  DOMAIN  TAXONOMY  MODELS.  Each  learning  domain  taxonomy 

model  consists  of  a  set  of  categories  into  which  performance  objectives, 
enabling  objectives,  and  prerequisites  may  be  grouped.  In  each  taxonomy 
model,  the  categories  are  mutually  exclusive  and  jointly  exhaustive,  i.e., 
every  objective  and  prerequisite  belongs  to  one  and  only  one  category  of 
each  taxonomy  model.  In  every  model,  the  categories  relate  in  some 
fashion  to  mental,  physical  and  attitudinal  actions/behaviors. 

The  three-category  taxonomy  model  (after  Bloom,  et  a!.): 

1.  Cognitive  domain. 

2.  Affective  domain. 

3.  Psychomotor  domain. 

The  four-category  taxonomy  model  (Branson,  et  al.): 

1 .  Information  domain  . 

2.  Mental  skills  domain. 

3.  Physical  skills  domain. 

4.  Attitudinal  domain. 

The  five-category  taxonomy  model  (Gagne,  et  al.): 

1.  Information  learning  domain. 

2.  Intellectual  skills  domain. 

3.  Cognitive  strategies  domain. 

4.  Motor  skills  domain. 

5.  Attitudinal  domain. 

See  also:  DOMAINS  OF  LEARNING 

LEARNING-HELPING.  Learning-helping  is  the  secondary,  supportive  function 
of  a  training  system. 

Learning-helping  is  any  activity  undertaken  to  provide  an  efficient 
learning  environment  for  the  learners. 

All  training  system  personnel  contribute  to  the  learning-helping  function. 

See  also:  LEARNING 

TRAINING  SYSTEM 

LEARNING  HIERARCHY.  The  learning  hierarchy  of  a  given  objective  is  an 
arrangement  of  its  essential  prerequisites  in  a  format  identifying  the 
sequencing  of  instruction  for  that  objective. 

See  also:  SEQUENCING  OBJECTIVES 
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LEARNING  OBJECTIVE. 


See:  PERFORMANCE  OBJECTIVE 

LEARNING  STYLE.  A  particular  learner's  learning  style  is  that  type  of 

sensory  activity  that,  typically,  is  the  best  way  for  him  or  her  to  learn 
(examples:  reading,  listening,  audio- visuals,  etc.). 

See  also:  INTERNAL  CONDITIONS  OF  LEARNING 

LEARNING  TASK  ANALYSIS.  A  component  of  task  analysis,  learning  task 

analysis  is  applied  to  a  performance  objective  and  to  its  mental  operations 
to  reveal  the  prerequisites  of  learning  and  the  desirable  sequence  of 
learning  events. 

The  focus  of  learning  task  analysis  is  on  the  prerequisites  for  learning 
the  task/performance  objective. 

Learning  task  analysis  pertains  both  to  the  present  conditions  for 
learning  and  to  the  prior  effects  of  learning. 

See  also:  TASK  ANALYSIS 

INFORMATION  PROCESSING  ANALYSIS 
TASK  CLASSIFICATION 

LESSON  PLAN.  A  lesson  plan  is  a  written  outline  of  the  instruction  to  be 

conducted  to  help  the  learner  achieve  a  particular  performance  objective 
or  enabling  objective. 

Steps  in  lesson  planning: 

1 .  Identify  the  perfor:  lance  objective  or  enabling  objective  for  which 
the  lesson  plan  is  to  be  written. 

2.  List  the  desired  instructional  events. 

3.  Select  the  ideal  media. 

4.  Select  the  materials  and  activities. 

5.  Analyze  the  materials  to  identify  the  events  they  supply. 

6.  Plan  other  means  of  implementing  the  remaining  events. 

See  also:  STRATEGY  OF  INSTRUCTION 
TACTICS  OF  INSTRUCTION 
INSTRUCTIONAL  EVENTS 

LEVELS  OF  OBJECTIVES.  The  levels  of  objectives  form  a  hierarchical 

arrangement  beginning  with  the  most  general  definition  of  the  ultimate 
intended  outcomes  of  learning,  and  proceeding  to  the  intended 
achievements  associated  with  the  smallest  units  of  instruction. 

Seven  levels  of  objectives  are  defined: 

1.  Needs. 

2.  Goals. 

3.  Life-long  objectives. 
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4.  End-of-course  objectives. 

5.  Units  of  objectives. 

6.  Performance  objectives. 

7.  Enabling  objectives. 

See  also:  NEED 
GOAL 

PERFORMANCE  OBJECTIVE 
ENABLING  OBJECTIVE 

LEVELS  OF  SYSTEM  DESCRIPTION.  The  levels  of  human-machine  system 
description  form  one  dimension  for  organizing  the  system's  work 
requirements.  The  levels  of  description  address  the  work,  itself,  why 
the  work  is  required,  and  how  its  requirements  are  to  be  met. 

The  levels  of  system  description  include  three  hierarchically  related 
categories: 

1 .  Objectives  level 

(What  work  is  required,  and  what  impediments  the  work  faces.) 

2.  Functional  Purposes  level 

(Why  each  Objectives  level  requirement  exists,  and  the 
impediments  that  apply  to  satisfying  each  purpose.) 

3.  Characteristics  level 

(How  each  Functional  Purposes  level  requirement  will  be  met, 
and  the  impediments  that  apply  to  carrying  out  each 
Characteristics  step.) 

The  levels  of  system  description  form  one  dimension  of  the  Systems 
Taxonomy  Model.  They  interact  with  another  dimension,  viz.,  aspects  of 
performance,  to  form  fifteen  cells  or  sub-taxonomies  within  which  any 
system's  work  requirements  can  be  exhaustively  identified  and  organized. 

See  also:  SYSTEMS  TAXONOMY  MODEL 
ASPECTS  OF  PERFORMANCE 
OBJECTIVES-LEVEL  TAXA 
FUNCTIONAL  PURPOSES-LEVEL  TAXA 
CHARACTERISTICS-LEVEL  TAXA 

LOGISTICS  SUBSYSTEM.  The  Logistics  Subsystem  is  one  of  the  six  generic, 
functionally  oriented  subsystems  of  any  training  system.  Logistics 
Subsystem  deals  with  maintenance  of  facilities  and  equipment;  housing, 
feeding  and  recreation  of  training  system  personnel;  replenishment  of 
consumables;  transportation  of  people  and  materials;  general 
housekeeping;  etc. 

Logistics  contributes  to  the  learning-helping  function  by  attending  to  the 
myriad  of  details  necessary  to  keep  the  training  system  running  smoothly 
and  free  of  discomfort/distraction. 
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The  principal  operator  or  staff  member  of  the  Logistics  Subsystem  is  the 
logistics  supporter. 

See  also:  TRAINING  SYSTEM 
LEARNING 
LEARNING-HELPING 

MEASURE.  A  measure  is  a  judgment  or  appraisal  of  some  attribute  of  a 
system. 

A  measure  is  a  test  or  assessment  of  whether  the  attribute  is  "good 
enough"  to  meet  the  demands  of  the  performance  requirement  or  taxon  to 
which  the  attribute  is  linked. 

Every  measure  contributes  an  important  bit  of  information  about  the 
system's  satisfaction  of  the  performance  taxon  of  interest. 

See  also:  ATTRIBUTE 
TAXON 

DESIGN  SPECIFICATION 
MEASUREMENT. 

See:  EVALUATION 
MEASUREMENT  APPLICATION. 

See:  MEASUREMENT  PURPOSE 

MEASUREMENT  PURPOSE.  The  measurement  purpose  for  a  particular’ 

application  of  the  Analytic  Process  Model  to  some  system  is  the  specific 
work  of  that  system  that  is  to  be  evaluated  in  the  application. 

A  measurement  purpose  is  equivalent  to  the  particular  sub-taxonomy  of 
system  performance  requirements  for  which  measures  are  to  be  selected 
and  applied. 

Synonym:  MEASUREMENT  APPLICATION 

See  also:  TAXONOMY 
MEASURE 

MENTAL  SKILLS  DOMAIN  OBJECTIVES.  Performance  objectives  belonging  to 
the  mental  skills  domain  require  that  the  learner  demonstrate  the  ability 
to  identify,  classify,  use  rules,  or  solve  problems  involving  thinking, 
creating  and  analyzing. 

See  also:  PERFORMANCE  OBJECTIVES 

LEARNING  DOMAIN  TAXONOMY  MODELS 
PHYSICAL  SKILLS  DOMAIN  OBJECTIVES 
INFORMATION  DOMAIN  OBJECTIVES 
ATTITUDINAL  DOMAIN  OBJECTIVES 
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NEED.  A  need  is  a  discrepancy  or  gap  between  the  way  things  are  and  the 
way  things  ought  to  be. 

A  need  exists,  for  example,  when  a  person  who  cannot  presently  perform 
a  particular  job  wishes  or  is  expected  to  perform  that  job. 

See  also:  NEEDS  ASSESSMENT 

TRAINING  NEEDS  ASSESSMENT 
GOAL 

NEEDS  ASSESSMENT.  A  systematic  process  for  determining  performance  goals, 
identifying  discrepancies  between  goals  and  the  current  status  of 
performance,  and  establishing  priorities  for  action. 

T.t  should  not  be  assumed  that  all  needs  (performance  discrepancies)  can 
be  solved  through  training.  Some  arise  due  to  external  conditions. 

Needs  assessment  enables  the  curriculum  developers  to  .  focus  on 
high-priority  training  needs. 

See  also:  NEED 
GOAL 

TRAINING  NEEDS  ASSESSMENT 
PROBLEM  ANALYSIS 

OBJECTIVES-LEVEL  TAXA.  Objectives-level  taxa  are  statements  of  what 

performance  is  required  of  a  system,  without  reference  to  why  the 
performance  is  required  or  how  it  is  required  to  be  performed. 

Objectives-level  taxa  include  definitions  of  the  basic  capabilities, 
activities,  and  outputs  expected  of  the  system  and  of  the  relevant 
environmental  and  constraining  factors  that  apply  to  those  basic 
capabilities,  activities  and  outputs. 

Objectives-level  taxa  consist  of  the  system's  performance  potentialities, 
processes,  products,  environment  and  constraints  viewed  at  the  topmost 
level  of  system  description.  It  is  a  view  of  work  requirements  as  objects 
for  study  in  and  of  themselves. 

Cpp  ol cn •  T AXON 

FUNCTIONAL  PURPOSES-LEVEL  TAXA 
CHARACTERISTICS-LEVEL  TAXA 
LEVELS  OF  SYSTEM  DESCRIPTION 
ASPECTS  OF  PERFORMANCE 

OBJECTIVE-REFERENCED  TEST.  A  test  designed  to  determine  a  learner's 

achievement  of  a  specified  performance  objective,  enabling  objective,  or 
prerequisite. 

An  objective-referenced  test  must  be  keyed  to  the  statement  of  the 
objective,  and  especially  to  the  capability  verb  and  the  action  verb  in  the 
statement. 
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The  capability  verb  indicates  the  domain  of  learning  to  be  addressed  by 
the  test.  The  action  verb  indicates  the  exact  behavior  that  the  test  must 
elicit. 

See  also:  ENTRY  TEST 
PRETEST 
POSTTEST 

PEER  TUTORING.  Peer  tutoring  is  a  training  procedure  in  which  one  learner 
first  observes,  then  practices  and  learns  a  task,  and  then  demonstrates 
the  task  to  another  learner. 

PERFORMANCE  CONSTRAINTS.  A  system's  performance  constraints  are  the 
artificially  imposed  factors  governing  the  circumstances  and  situations  of 
the  system's  operation  that  may  affect  whether  and  how  the  system 
satisfies  its  work  requirements. 

Constraints  include  all  man-made  impediments  to  the  system's  potential, 
process  or  product  performance. 

See  also:  ASPECTS  OF  PERFORMANCE 
PERFORMANCE  ENVIRONMENT 
PERFORMANCE  POTENTIALITIES 
PERFORMANCE  PROCESSES 
PERFORMANCE  PRODUCTS 
LEVELS  OF  SYSTEM  DESCRIPTION 

PERFORMANCE  ENVIRONMENT.  A  system's  performance  environment  is  the  set 
of  naturally  occurring  factors  associated  with  or  incidental  to  the 
circumstances  and  situations  of  the  system's  operation  that  may -affect  the 
system's  satisfaction  of  its  work  requirements. 

The  environment  includes  all  natural  impediments  to  the  system's 
potential,  process,  or  product  performance. 

See  llso:  ASPECTS  OF  PERFORMANCE 
PERFORMANCE  CONSTRAINTS 
PERFORMANCE  POTENTIALITIES 
PERFORMANCE  PROCESSES 
PERFORMANCE  PRODUCTS 
LEVELS  OF  SYSTEM  DESCRIPTION 

PERFORMANCE  OBJECTIVE.  Performance  objectives  are  written  explicit 

descriptions  of  specific  terminal  behaviors  or  instructional  outcomes 
required  of  learners  to  signify  successful  completion  of  their  study. 

Performance  objectives  are  statements  of  what  the  learner  will  be  able  to 
do,  or  how  the  learner  will  be  expected  to  behave,  after  completing 
instruction . 

Performance  objectives  identify  the  end  products  of  instruction  in  terms 
of  observable,  measurable  behavior. 
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The  objective  is  a  specific  description  of  the  action  the  learner  is  to 
exhibit  after  training,  the  conditions  under  which  the  action  will  take 
place,  and  the  standard  or  criterion  which  must  be  reached  for 
satisfactory  performance. 

The  objective  may  state  a  directly  observable  action,  or  an  observable 
product  resulting  from  the  action. 

Performance  objectives  must  clarify,  in  unambiguous  terms,  the  behavioral 
competencies  to  be  developed  by  the  learner. 

Potential  errors/defects  that  may  exist  in  statements  of  performance 
objectives: 

1.  They  may  refer  to  learners’  mental  processes  rather  than  to 
observable  actions  or  behaviors. 


2.  The  performance  standard  may  be  missing  or  deficient. 

3.  The  conditions  may  not  be  stated  or  may  be  poorly  specified. 

4.  The  objective  may  state  what  the  facilitator  does  rather  than  what 
the  learner  will  do. 

5.  The  stated  behavior  may  be  too  broad  to  describe  specific  intended 
outcomes  adequately. 

Synonyms:  LEARNING  OBJECTIVE 

BEHAVIORAL  OBJECTIVE 

See  also:  PERFORMANCE  OBJECTIVE  MODELS 
ENABLING  OBJECTIVE 
LEVELS  OF  OBJECTIVES 
LESSON  PLAN 
TASK 

PERFORMANCE  OBJECTIVE  MODELS.  The  three-component  model  (after 
Branson,  Mager,  et  al . ) : 

1.  Statement  of  the  action  involved  in  the  performance. 

2.  Statement  of  the  conditions  under  which  the  action  will  take  place. 

3.  Statement  of  how  well  the  action  must  be  performed. 

The  five-component  model  (after  Gagne,  Briggs,  et  al.): 

1.  Action — something  observable  that  the  learner  will  be  doing. 

2.  Object- -something  concrete  that  the  learner  will  produce  via  the 
action. 

3.  Situation — the  "givens." 
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4.  Tools  and  constraints — applicable  to  the  user's  implementation  of  the 
action. 

5.  The  learning  capability--that  the  action/object  indicates  has  been 
acquired. 

See  also:  PERFORMANCE  OBJECTIVE 
ENABLING  OBJECTIVE 

PERFORMANCE  POTENTIALITIES.  A  system's  performance  potentialities  are 
those  taxa  of  required  performance  that  define  the  basic,  inherent 
capabilities  that  the  system  is  to  possess. 

See  also:  TAXON 

PERFORMANCE  PROCESSES 
PERFORMANCE  PRODUCTS 
ASPECTS  OF  PERFORMANCE 
LEVELS  OF  SYSTEM  DESCRIPTION 

PERFORMANCE  PROCESSES.  A  system's  performance  processes  are  those  taxa 
of  required  performance  that  define  the  activities  the  system  is  to  carry 
out,  or  the  methods/techniques  the  system  is  to  employ. 

See  also:  TAXON 

PERFORMANCE  POTENTIALITIES 
PERFORMANCE  PRODUCTS 
ASPECTS  OF  PERFORMANCE 
LEVELS  OF  SYSTEM  DESCRIPTION 

PERFORMANCE  PRODUCTS.  A  system's  performance  products  are  those  taxa 
of  required  performance  that  define  the  goods,  services  or  other  output 
that  the  system  is  to  deliver. 

See  also:  TAXON 

PERFORMANCE  POTENTIALITIES 
PERFORMANCE  PROCESSES 
ASPECTS  OF  PERFORMANCE 
LEVELS  OF  SYSTEM  DESCRIPTION 

PHYSICAL  SKILLS  DOMAIN  OBJECTIVES.  Performance  objectives  belonging  to 
the  physical  skills  domain  require  that  the  learner  demonstrate  the  ability 
to  carry  out  some  physical,  manipulative  action. 

See  also:  PERFORMANCE  OBJECTIVES 

LEARNING  DOMAIN  TAXONOMY  MODELS 
INFORMATION  DOMAIN  OBJECTIVES 
MENTAL  SKILLS  DOMAIN  OBJECTIVES 
ATTITUDINAL  DOMAIN  OBJECTIVES 

PILOT  TESTING.  Pilot  testing  is  the  delivery  of  a  newly  developed  program 
of  instruction,  in  the  actual  intended  operational  environment,  for  the 
purpose  of  evaluating  and  improving  the  program. 
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Synonym:  FIELD  TRIAL 

See  also:  FORMATIVE  EVALUATION 
FEEDBACK 

POSTTEST.  A  posttest  is  a  test  of  a  performance  objective  or  enabling 

objective,  administered  subsequent  to  delivery  of  prescribed  training  for 
that  objective. 

Posttests  provide  the  basis  for  determining  whether  particular  learners 
require  additional  training,  or  whether  they  can  be  certified  as  having 
achieved  a  given  set  of  performance  objectives. 

See  also:  OBJECTIVE-REFERENCED  TEST 
ENTRY  TEST 
PRETEST 

PREDICTIVE  VALIDITY  OF  TRAINING.  The  predictive  validity  of  the  training 
undertaken  to  pursue  a  particular  performance  objective  is  the  correlation 
between  a  learner's  posttest  score  on  that  objective  and  the  learner's 
subsequent  job  performance  measure  on  the  corresponding  task. 

See  also:  POSTTEST 

JOB  PERFORMANCE  MEASURE 
PERFORMANCE  OBJECTIVE 

PREREQUISITE.  A  prerequisite  is  a  performance  objective  which  candidate 
learners  are  expected  to  have  achieved  prior  to  the  prescribed  training. 

See  also:  PERFORMANCE  OBJECTIVE 
ESSENTIAL  PREREQUISITE 
SUPPORTIVE  PREREQUISITE 
ENTRY  TEST 

PRESENT  CONDITION  OF  LEARNING.  The  present  condition  of  learning  is 

that  which  the  learner  must  recall  to  make  achievement  of  the  particular 
performance  objective  possible. 

The  present  condition  of  learning  is  revealed  by  Learning  Task  Analysis. 

See  also:  LEARNING  TASK  ANALYSIS 

INTERNAL  CONDITIONS  OF  LEARNING 

PRETEST.  A  pretest  is  a  test  of  a  performance  objective  or  enabling 

objective,  administered  prior  to  delivery  of  the  prescribed  training  for 
that  objective. 

Pretests  provide  the  basis  for  determining  whether  particular  learners 
should  "skip"  blocks  of  instruction  corresponding  to  objectives  already 
achieved. 

See  also:  OBJECTIVE-REFERENCED  TEST 
ENTRY  TEST 
POSTTEST 
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PROBLEM  ANALYSIS.  Given  evidence  of  a  failure  or  discrepancy  in 

performance,  problem  analysis  is  conducted  to  determine  whether  the 
problem/need  should  be  solved  by  means  of  training,  equipment, 
administrative  action,  or  some  other  method. 

See  also:  NEEDS  ASSESSMENT 

PROMPTING.  Prompting  is  providing  direct  guidance  to  a  learner  by 

supplying  a  specific  fact,  procedure,  word,  phrase,  etc.,  that  the 
learner  is  attempting  to  recall. 

See  also:  GENERAL  LEARNING  GUIDELINES 
CUE 

PURPOSES  OF  AN  INSTRUCTIONAL  UNIT.  Identification  of  the  purposes  of 
an  instructional  unit  provides  a  first  step  toward  defining  the 
performance  objectives  of  that  unit. 

The  stated  instructional  purposes  must: 

1.  Identify  what  learners  will  do  after  or  as  a  result  of  the  instruction, 
not  what  they  will  do  during  the  instruction. 

2.  Identify  current  or  immediate  outcomes  expected  of  the  instruction, 
not  long-range,  future  outcomes. 

REMEDIAL  TRAINING.  The  remedial  training  conducted  in  support  of  a 

particular  program  of  instruction  is  the  training  given  to  certain 
candidate  learners  to  help  them  achieve  performance  objectives  that  are 
considered  prerequisites  for  the  program  of  instruction. 

See  also:  ENTRY  TEST 

ESSENTIAL  PREREQUISITE 
SUPPORTIVE  PREREQUISITE 

SEQUENCING  INSTRUCTION. 

See:  SEQUENCING  OBJECTIVES 

SEQUENCING  OBJECTIVES.  Sequencing  objectives  is  the  process  of 

determining  the  order  in  which  performance  objectives  and  enabling 
objectives  will  be  pursued  during  training.  Proper  sequencing  demands 
that  objectives  be  pursued  in  a  logical  "stepping-stone"  fashion. 
Specifically,  an  objective  should  not  be  pursued  until  those  that  are 
essential  or  helpful  for  its  pursuit  have  first  been  achieved. 

In  sequencing  objectives  for  pursuit  during  training: 

1.  Place  dependent  objectives  after  those  on  which  they  depend. 

2.  Of  two  supportive  objectives,  the  one  that  is  easier/cheaper  to 
achieve  should  be  placed  first. 
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Possible  errors  that  may  occur  in  sequencing  objectives: 

1 .  A  supportive  relationship  could  be  misinterpreted  as  a  dependent 
relationship,  and  result  in  increased  training  time  and  expense. 

2.  A  dependent  relationship  could  be  misinterpreted  as  independent, 
and  result  in  a  very  difficult-to-achieve  sequence. 

3;  A  supportive  relationship  might  not  be  recognized,  and  result  in 
inefficient  instruction. 

Synonym:  SEQUENCING  INSTRUCTION 

Sec  also:  DEPENDENT  OBJECTIVE 
SUPPORTIVE  OBJECTIVES 

STIMULUS  OF  A  TASK.  The  stimulus  of  a  task  is  that  which  "triggers"  the 
performance  of  the  task  by  the  job  performer. 

STIMULUS  OF  INFORMATION  PRESENTATION.  The  stimulus  for  a  particular 
block  of  information  is  the  form  in  which  that  block  is  presented  to  the 
learner. 


STM. 


See:  SYSTEMS  TAXONOMY  MODEL 

STRATEGY  OF  INSTRUCTION.  The  strategy  of  instruction  for  a  given 

performance  objective  consists  of  1)  determining  the  sequence  of  pursuit 
of  enabling  objectives  for  that  performance  objective,  and  2)  choosing  how 
to  implement  instructional  events  for  each  enabling  objective. 

See  also:  LESSON  PLAN 

TACTICS  OF  INSTRUCTION 

SUBSYSTEM.  A  subsystem  of  a  particular  human-machine  system  of  interest 
is,  itself,  a  human-machine  system  that  is  subservient  to  the  system  of 
interest  in  the  work  it  performs. 

Subsystems  may  be  totally  contained  within  the  system  of  interest. 

Subsystems  are  members  of  the  system  hierarchy  of  the  system  of 
interest. 

See  also:  HUMAN-MACHINE  SYSTEM 
SYSTEM  HIERARCHY 

SUMMATIVE  EVALUATION.  Summative  evaluation  is  the  process  of  gathering, 
combining  and  interpreting  data  to  measure  the  effectiveness  of  a  training 
system  operating  at  full-scale  delivery. 
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Summative  evaluation  occurs  after  all  formative  evaluation  has  been 
completed. 

See  also:  FORMATIVE  EVALUATION 

SUPPORTIVE  OBJECTIVES.  Two  performance  objectives,  or  two  enabling 
objectives,  arc  supportive  of  one  another  when  achievement  of  one 
facilitates  achievement  of  the  other,  regardless  of  the  order  of 
achievement. 

If  two  objectives  are  supportive  of  one  another,  the  one  that  is 
easier /cheaper  to  achieve  should  be  pursued  first,  during  training. 

See  also:  DEPENDENT  OBJECTIVE 

INDEPENDENT  OBJECTIVES 
PERFORMANCE  OBJECTIVE 
SEQUENCING  OBJECTIVES 

SUPPORTIVE  PREREQUISITE.  A  supportive  prerequisite  for  a  particular 

performance  objective  or  enabling  objective  is  a  performance  capability 
that  is  not  actually  incorporated  within  the  objective,  but  which  makes  it 
easier  to  learn/achieve  the  objective. 

Learners  who  possess  the  supportive  prerequisite  will  usually  achieve  the 
objective  more  readily  than  will  learners  who  do  not  possess  it. 

See  also:  PERFORMANCE  OBJECTIVE 
ENABLING  OBJECTIVE 
SUPPORTIVE  OBJECTIVES 
DEVELOPMENT  OF  OBJECTIVES 
ESSENTIAL  PREREQUISITE 

SUPRASYSTEM.  A  suprasystem  of  a  particular  human-machine  system  of 

interest  is,  itself,  a  human-machine  system  to  which  the  system  of 
interest  is  subservient  in  the  work  it  performs. 

The  system  of  interest  may  be  totally  contained  within  the  suprasystem. 

Suprasystems  are  members  of  the  system  hierarchy  of  the  system  of 
interest. 

See  also:  HUMAN-MACHINE  SYSTEM 
SYSTEM  HIERARCHY 


SYSTEM. 

See:  HUMAN-MACHINE  SYSTEM 

SYSTEM  HIERARCHY.  The  system  hierarchy  surrounding  a  particular 

human-machine  system  of  interest  is  the  network  of  other  systems  with 
which  the  system  of  interest  interacts  in  the  performance  of  its  work. 
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System  hierarchy  is  one  dimension  of  the  Systems  Taxonomy  Model. 

Sec  also:  HUMAN-MACHINE  SYSTEM 
COLLATERAL  SYSTEM 
SUBSYSTEM 
SUPRASYSTEM 

SYSTEMS  TAXONOMY  MODEL 

SYSTEMS  TAXONOMY  MODEL.  The  Systems  Taxonomy  Model  is  a  conceptual 
framework  and  a  set  of  analytic  procedures  for  identifying  the 
performance  requirements  (or  taxa)  associated  with  a  system  of  interest. 

The  Systems  Taxonomy  Model  is  the  first  stage  of  the  Analytic  Process 
Model. 

The  Systems  Taxonomy  Model  identifies  and  organizes  the  performance 
taxa  of  a  system  of  interest  along  three  dimensions: 

1.  System  Hierarchy — This  dimension  insures  that  work  requirements 
will  be  identified  by  studying  the  system  of  interest,  itself,  as  well 
as  by  studying  the  work  requirements  imposed  upon  the  system  of 
interest  by  other,  interacting  systems  that  are  linked  to  it 
hierarchically. 

2.  Aspects  of  Performance — This  dimension  insures  that  the  identified 
work  requirements  will  include  not  only  what  the  system  is  to 
produce,  but  also  the  processes  it  is  to  carry  out,  the  capabilities  it 
is  to  possess,  and  the  circumstances  under  which  it  is  to  operate. 

3.  Levels  of  System  Description— -This  dimension  insures  that  the  work 
requirements  will  be  identified  not  only  in  terms  of  what  is  expected, 
but  also  why  it  is  needed  and  how  it  is  supposed  to  be 
accomplished. 

Synonym:  STM 

See  also:  ANALYTIC  PROCESS  MODEL 
TAXON 
TAXONOMY 

ASPECTS  OF  PERFORMANCE 
LEVELS  OF  SYSTEM  DESCRIPTION 
SYSTEM  HIERARCHY 

TACTICS  OF  INSTRUCTION.  Tactics  of  instruction  are  the  specific  details  or 
prescriptions  for  the  instructional  events  in  a  lesson  plan,  with  special 
reference  to  how  the  needed  conditions  of  learning  are  incoprorated  into 
the  events. 

See  also:  LESSON  PLAN 

STRATEGY  OF  INSTRUCTION 
CONDITIONS  OF  LEARNING 
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TASK.  A  task  is  an  element  of  a  job.  A  task  can  be  described  in  terms  of  a 
discrete,  observable  action  carried  out  by  the  job  performer.  Every  task 
must  be  addressed  in  the  training  provided  for  the  job.  Any  given  task 
may  be  represented  by  a  performance  objective,  an  enabling  objective,  or 
a  prerequisite. 

See  aiso:  JOB  ANALYSIS 
TASK  ANALYSIS 
PERFORMANCE  OBJECTIVE 

TASK  ANALYSIS.  The  analysis  of  an  individual  task,  or  performance 
objective,  to  provide  a  basis  for  determining  the  appropriate 
teaching-learning  strategy  for  that  task/objective. 

Task  analysis  has  three  components: 

1.  Information  processing  analysis. 

2.  Task  classification. 

3.  Learning  task  analysis. 

Task  analytic  products: 

1.  information-processing  diagram  showing  the  mental  operations 
involved  in  each  performance  objective. 

2.  Classification  of  each  objective  into  the  appropriate  class  of  learning 
outcome,  or  domain  of  learning. 

3.  Specification  of  essential  and  supportive  prerequisites  for  each 
performance  objective. 

Synonym:  ANALYSIS  OF  PERFORMANCE  OBJECTIVES 

TASK  CLASSIFICATION.  A  component  of  Task  Analysis,  task  classification 
categorizes  a  performance  objective  as  to  its  domain  of  learning. 

Task  classification  provides  a  means  of  identifying  the  necessary 
conditions  of  learning  that  are  to  be  included  in  instructional  events. 

See  also:  TASK  ANALYSIS 

INFORMATION  PROCESSING  ANALYSIS 
LEARNING  TASK  ANALYSIS 
LEARNING  DOMAIN  TAXONOMY  MODELS 

TAXON.  A  particular  category  or  class  pertaining  to  some  thing  or  concept  of 
interest. 

A  taxon  is  an  element  of  a  taxonomy. 

A  system  performance  taxon  is  a  particular  type  or  category  of 
performance  that  is  required  of  a  system  of  interest. 
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See  also:  TAXONOMY 

SYSTEMS  TAXONOMY  MODEL 
ASPECTS  OF  PERFORMANCE 
LEVELS  OF  SYSTEM  DESCRIPTION 

TAXONOMY.  (When  written  with  an  initial  capital  "T"):  The  science  or 
technique  of  classification,  as  in  the  technique  ot  classifying  and 
organizing  the  performance  requirements  of  a  human-machine  system. 

(When  written  with  a  lower  case  ”t"):  A  particular  classification  of 
things  or  concepts,  as  in  a  classification  and  organization  of  a  particular 
system's  performance  requirements. 

See  also:  TAXON 

SYSTEMS  TAXONOMY  MODEL 


TEST. 

See:  OBJECTIVE-REFERENCED  TEST 

TEST  DISTORTION.  Test  distortion  is  a  defect  in  a  test  resulting  from  the 
fact  that  the  test  can  be  answered/performed  successfully  by  applying  a 
domain  of  learning  and/or  action  other  than  that  intended  to  be  tested. 

See  also:  OBJECTIVE-REFERENCED  TEST 
TEST  RELIABILITY 
TEST  VALIDITY 

TEST  RELIABILITY.  The  reliability  of  a  test  is  assessed  by  examining 

whether  the  test  requires  the  learner  to  perform  the  action  verb  long 
enough/often  enough  to  permit  a  dependable  inference  that  the  objective 
has  been  demonstrated. 

See  also:  OBJECTIVE-REFERENCED  TEST 
ACTION  VERB 
TEST  DISTORTION 
TEST  VALIDITY 


TEST  VALIDITY.  The  validity  of  a  test  is  assessed  by  comparing  the  test 
with  the  objective  it  is  intended  to  measure.  The  action,  situation, 
standards,  etc.,  of  the  test  must  conform  exactly  to  those  of  the 
objective. 

See  also:  OBJECTIVE-REFERENCED  TEST 
PERFORMANCE  OBJECTIVE 
TEST  DISTORTION 
TEST  RELIABILITY 

TRAINING  NEEDS  ASSESSMENT.  Training  needs  assessment  is  a  process  for 
determining  what  should  be  taught/learned.  The  process  is  based  on  the 
active  involvement  of  those  directly  affected  by  the  training  system 
(e.g.,  the  learners),  in  addition  to  professional  educators. 
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Training  needs  assessment  consists  of: 

1.  Analysis  of  the  job  to  be  trained. 

2.  Selection  of  *he  tasks  to  be  trained. 

3.  Construction  of  job  performance  measures  for  the  selected  tasks. 

4.  Analysis  of  existing  courses. 

5.  Selection  of  the  instructional  setting  for  each  task  to  be  trained. 

See  also:  ANALYSIS 
NEED 

NEEDS  ASSESSMENT 

TRAINING  SYSTEM.  A  training  system  is  a  particular  type  of  human-machine 
system,  intended  to  accomplish  two  major  functions:  learning,  and 
learning-helping . 

Training  systems  have  six  types  of  operators,  or  staff  members: 
learners,  facilitators,  administrators,  curriculum  developers,  facilities 
developers,  and  logistic  supporters. 

Training  systems  have  six  functionally  oriented  subsystems,  each  of  which 
is  operated  principally  by  one  of  the  types  of  operators: 

Command  (training  administrator) 

Design  (curriculum  developer) 

Emplacement  (facilities  developer) 

Logistics  (logistics  supporter) 

Enabling  (facilitator) 

Delivery  (learner) 

See  also:  LEARNING 

LEARNING-HELPING 
COMMAND  SUBSYSTEM 
DESIGN  SUBSYSTEM 
EMPLACEMENT  SUBSYSTEM 
LOGISTICS  SUBSYSTEM 
ENABLING  SUBSYSTEM 
DELIVERY  SUBSYSTEM 
HUMAN-MACHINE  SUBSYSTEM 

TRAIT.  A  trait  is  a  qualitative  or  quantitative  property  (aptitude,  expertise, 
strength,  reliability,  etc.)  possessed  by  a  system  component  (person, 
equipment  or  procedure)  which  is  necessary  in  order  to  achieve  or  satisfy 
a  performance  taxon  of  interest.  The  presence  of  a  trait  is  determined 
by  one  or  more  concrete,  observable  attributes. 
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\PPENDIX  A 


Training  Design  Subsystem  Data  Base 
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rcuestonces  specified  for  execution  of  octions  by  gunner/learners 


3*2* 1.4. 1*1*  Specified  factors  not  applying  under  realistic  Job  conditions 
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Specification  of  special  location/equipeent  set-up  configuration  requirements 


3*3. 3.4. 0.0'  Specif icotion  of  instructional  personnel  needed 
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0*0*  Relating  the  learning  to  the  learners'  experience 


1 *3*3*1 *0*0*  Exploiting  learner*'  experiences  through  discussions*  case-studies*  role-play 
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5*2* 1*0*0*  Understanding  the  topics  to  be  covered 


I»3*2*2«0t0«  Understanding  the  associations  among  topics  and  performance  objectives 
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1.0. 0.0.  Facilitators  demonstrate  satsfactory  levels  of  Job  performance  sKills 
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3. 2* 1.1 *0.0.  Performance  objective*  spec*  stated  in  terms  relevant  to  learners 
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APPENDIX  C 


Training  Delivery  Subsystem  Data  Base 
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